Intel 9th Gen Paid Benchmarks Take Advantage of NDA Periods

[Carclis]

Intel has recently revealed it's latest 9th generation line of products and subsequently opened up pre-orders for them. As seems to be typical in the tech space there were no benchmarks available with all of the tech journalists being held back by NDA periods. There was one exception however; a publication known as PCGamesN who published benchmarks conducted by Principled Technologies under commission by Intel. The results in question proposed that the new 9900k CPU was up to 50% faster than the Ryzen 7 2700X in gaming. Steven Walton of TechSpot has expressed his concerns over this and also pointed out numerous issues with the testing done in the report.

Quote

Right away many of the results looked very suspect to me, having spent countless hours benchmarking both the 2700X and 8700K, I have a good idea of how they compare in a wide range of titles and these results looked very off. Having spotted a few dodgy looking results my next thought was, why is PCGamesN publishing this misleading data and why aren’t they not tearing the paid benchmark report apart? Do they simply not know better?

After doing his own research he found that numerous and seemingly intentional steps were taken in order to handicap the Ryzen system and present the new Intel product in the best possible light. The comparisons were clearly not apples to apples and the problems only just begin with their method of applying XMP.

Quote

they used Corsair Vengeance memory without loading the extreme memory profile or XMP setting, instead they just set the memory frequency to 2933 and left the ridiculously loose default memory timings in place. These loose timings ensure compatibility so systems will boot up, but after that point you need to enable the memory profile. It’s misleading to conduct benchmarks without executing this crucial step.

Steve's testing found the results contradictory to his own testing when which he repeated for the sole purpose of trying to replicate the Principled Technology results. His results for the Intel systems were not far off sans 9th series parts which are still pending the NDA lift but the Ryzen system was quite far off as you can see below.

For me this news is quite concerning. I'm not a fan of pre-orders, especially when no benchmarks are available. But to have commissioned benchmarks published during an NDA is a step too far and extremely deceptive.

Update
Intel Statement to Tom'sHardware

Quote

We are deeply appreciative of the work of the reviewer community and expect that over the coming weeks additional testing will continue to show that the 9th Gen Intel Core i9-9900K is the world’s best gaming processor. Principled Technologies conducted this initial testing using systems running in spec, configured to show CPU performance and has published the configurations used. The data is consistent with what we have seen in our labs, and we look forward to seeing the results from additional third-party testing in the coming weeks.

 

Angry Steve's take
Beware, you'll probably want to grab some popcorn for these.

 

The laundry list of issues

• Benchmarks were released during the NDA period
• Multiple GPU's used (GPU boost disparity can occur between different quality silicon)
• Median results taken instead of average performance
• AMD Ryzen 7 2700x test system used the stock cooler with Intel and others using a higher end Noctua NH-U14S cooler
• Memory was set to 2666Mhz on Intel and 2933Mhz on AMD systems despite being a 3000Mhz kit
• XMP was only enabled on Intel systems with no manual tuning to prevent loose timings on the AMD system
• 64GB of RAM used to represent the "typical gamer"
• Uncharacteristic performance deficits on AMD system in Ashes of the Singularity benchmark which typically favours AMD
• Game Mode enabled on the Ryzen system (disabling 50% of the cores)
• FFIV demo invalid as a benchmark due to many technical issues including geometry culling problems
• GTAV settings not clarified. The paper claims presets were used but presets do not include the advanced graphical settings and change every time a hardware change is made

 

Follow-up on interview questions

GPU model clarification

Quote

Regarding which specific GTX 1080 Ti was used, Principled Technologies emailed to let us know that it was the Gigabyte Aorus GTX 1080 Ti. The specific model number is GV-N108TAORUS-11GD.

Number of test benches

Quote

As for clarification on the number of test benches, we learned post-interview that Principled Technologies used a total of sixteen systems – eight core test platforms, then eight duplicates. We are still uncertain as to if PT then took the median of all six test passes or focused on only three passes from a single system.

PT’s Response to GN Claims of Unfair Cooler Arrangement

Quote

We pressed a question pertaining to thermals repeatedly in the interview, found at mark 11:43, and expressed concern over the choice to use aftermarket coolers only on the Intel DT parts, but not AMD. AMD’s Ryzen 7 2700X CPU was left with its stock cooler, whereas all competing Intel parts used the Noctua NH-U14S, which is one of the best 140mm coolers we’ve worked with. Our chief concern stemmed from multiple thermal constraints in stacking to potentially inhibit AMD’s CPU performance, including usage of the thermally obstructed front panel of the Thermaltake Suppressor case. Forcing air to follow multiple 90-degree turns (and thus lose pressure) before it meets a downdraft cooler is already inadvisable. To exaggerate this effect by then using objectively better coolers on the competing Intel parts is disingenuous at worst, or inexperienced at best. We asked why the AMD unit used a stock cooler (see 11:43 in the video) and received the start of an answer in the video, but got the rest via email:

 

“Only the 2700X came with a cooler. The rest required a cooler to be added. AMD certainly did not seem to think their cooler was inadequate, so it seemed a reasonable choice.”

 

Of course, our argument is two-fold: As an independent test outlet, AMD’s word shouldn’t be good enough for anything – and AMD might not know that the cooler is in a suffocated case with low static pressure. Further, all of this is irrelevant: If testing in a controlled environment, the single element which matters is equality between all test beds. If that’s an NH-U14S, so be it – AMD gets one, too. In this instance, for whatever reason, it seems the AMD platforms were relegated to objectively weaker stock coolers, whereas Intel’s lack of a stock cooler has anointed its CPUs with high-end air coolers. AMD is being irresponsibly punished by including a cooler in its packaging, whereas Intel is rewarded with its lack of cooler by receiving a better solution. We are still uncertain as to what was used for the Threadripper parts, as the NH-U14S is natively incompatible without using the significantly altered TR4 variant of the cooler.

Memory Selection and 64GB Questions

Quote

The next follow-up point pertained to memory, chiefly the question of why 64GB was used. As our audience is likely aware, opting for higher memory capacities can result in lowered sustainable frequency and loosened timings – particularly tertiary and secondary, which were (as far as we’re aware) left uncontrolled by PT. In the on-camera interview (mark 19:20) we asked why 64GB was chosen, receiving the answer that it seems “normal” for the configurations. At best, this is disjointed from the buyer of the 2700X or 8700K CPUs. To purchase the kit of 64GB used in this test would cost $620, or around 2x the cost of the processors. Very few are reasonably purchasing this much memory and, given that this benchmark focuses entirely on gaming tests (and not “production” tasks), we must look at it from a gaming scenario. No meaningful gaming build with a non-HEDT platform is opting for 64GB of memory. Anyway, following the interview, we received additional information pertaining to memory kit selection:

 

“We chose Corsair memory due to past experience with it being very reliable. We chose the specific model to be able to get matching two 2x16GB packages in the quantity necessary for this project […] We want to minimize disk access while providing a RAM amount that seemed appropriate for high-end processors. While that might be a bit much for some of the slower processors, we did need to keep the playing field level.”

 

A few things to note here: Corsair isn’t really the reason memory is reliable, although that is a flattering statement to the company. Realistically, it’s the IC provider (and the PCB to some extent, but less so), and that’d be SK Hynix, Micron, or Samsung. Second, disk access from games will rarely exceed the hundreds of megabytes, much less 32GB or beyond. We’re talking about texture files and meshes and assets, most of which inevitably get shoved into GPU memory, anyway. System memory rarely exceeds 8-12GB utilization in even the most abusive of games. If recommending a gaming PC build, even if it hosts an HEDT CPU (because gaming is the only thing tested in the PT document), we would push for 32GB of RAM for easier-to-manage timings and frequencies. You can still get four sticks, so quad-channel operation remains feasible, and 32GB is still more than enough for gaming benchmarks. It’s more than enough for most benchmarks.

 

At some point, leveling the playing field is not the right move, anyway. If testing HEDT CPUs for production tasks, we must consider the element of realism and what the users will do with those parts. HEDT CPUs make more sense for pairing with 64GB in video production and production tasks, for instance. It is sometimes better to populate a chart with fewer devices that are more closely related.

 

As for why memory was manually downclocked to 2666MHz (Intel) and 2933MHz (AMD), the post-interview answer changed to the following: “We wanted to make sure none of the processors were overclocking. We declocked the RAM from a maximum of 3000 to what the vendor recommended.” We are not aware of instances where the CPU will self-overclock as a result of higher frequency memory, but it is more likely that PT was referring to the memory clocks being officially over “stock” CPU support levels. GamersNexus does not consider XMP to be “overclocking” for a 3000MHz kit, so we will agree to disagree on this point. No one buys 2666MHz kits. They’re the same price as 3000MHz kits (these days), and it’s a simple toggle to jump into full rated speeds.

GTA Quality Settings

Quote

We separately asked about GTA V quality settings, as these were left undefined in the initial testing document. We also raised concern of GTA’s propensity to change settings between hardware changes, which PT acknowledged. This was their answer:

 

GTA did have a strong tendency to try to reset its graphics settings, so we did have to carefully monitor that. Here is the output from the GTA benchmark text file:

 

Display: 1920x1080 (FullScreen) @ 60Hz VSync OFF

Tessellation: 0

LodScale: 0.000000

PedLodBias: 0.000000

VehicleLodBias: 0.000000

ShadowQuality: 1

ReflectionQuality: 1

ReflectionMSAA: 0

SSAO: 0

AnisotropicFiltering: 0

MSAA: 0

MSAAFragments: 0

MSAAQuality: 0

SamplingMode: 0

TextureQuality: 2

ParticleQuality: 0

WaterQuality: 0

GrassQuality: 0

ShaderQuality: 0

Shadow_SoftShadows: 0

UltraShadows_Enabled: false

Shadow_ParticleShadows: true

Shadow_Distance: 1.000000

Shadow_LongShadows: false

Shadow_SplitZStart: 0.930000

Shadow_SplitZEnd: 0.890000

Shadow_aircraftExpWeight: 0.990000

Shadow_DisableScreenSizeCheck: false

Reflection_MipBlur: true

FXAA_Enabled: false

TXAA_Enabled: false

Lighting_FogVolumes: true

Shader_SSA: false

DX_Version: 2

CityDensity: 0.000000

PedVarietyMultiplier: 1.000000

VehicleVarietyMultiplier: 1.000000

PostFX: 0

DoF: false

HdStreamingInFlight: false

MaxLodScale: 0.000000

MotionBlurStrength: 0.000000

Next Steps

Quote

Principled Technologies is actively working on a longer response to the most common concerns relating to its testing. We are uncertain as to when this will be published, but would expect its publication while the issue is still hot. PT Also alerted us that they are retesting most things, particularly highlighting that the company noticed greater “variability” with Game Mode on the 2700X. As Game Mode disables a CCX, it’d make sense that performance would be significantly handicapped on the AMD DT processor when running with half of its cores disabled.

 

For more discussion on this, view our embedded in-person interview above.

UPDATE: Principled Tech Publishes Comments

Quote

Principled Tech has published its own document commenting on current inquiries, found in full over here. We are rehosting it here for archival purposes. Key quotes are below:

"We have received a number of inquiries regarding the testing methodology we used and the potential for bias in favor of Intel. We are providing additional information to be as transparent as possible and to help allay these concerns.

"The following list summarizes many of the inquiries we have received and our responses. (We are continuing to work on addressing additional inquiries.)

• Use of "Game Mode" on the AMD Ryzen 7 2700X: Some inquiries we have received concern the use of the Ryzen utility and the number of active cores in the AMD-based systems. Based on AMD's recommendations and our initial testing on the Threadripper processors, we found installing the AMD Ryzen Master utility and enabling Game Mode increased most results. For consistency purposes, we did that for all AMD systems across Threadripper and Ryzen. We are now doing additional testing with the AMD systems in Creator Mode. We will update the report with new results.
• Cooler choice: We chose Noctua for the CPU coolers, due to having almost identical systems in the NH-U14S (Intel) and NH-U14S TR4-SP3 (AMD), which allowed us to maintain a comparable thermal profile. Because we were not performing any overclocking on any configuration, and because AMD has said it was a good cooler, we stuck with the stock AMD Ryzen 7 2700X Wraith Prism cooler.
• Memory speeds: To have complete parity across all systems, and to allow the Intel Core i9 X-series and AMD Ryzen Threadripper to fully utilize memory bandwidth, we used 4 16GB DDR4 DIMMs on all configurations.

[...]

"Because our goal is always to do the right thing and get the answers right, we are currently doing additional testing. We will share that data and will certainly call out if something is significantly different from what we've already published. 

"We are confident in our test methodology and results. We welcome questions and we are doing our best to respond to questions from our interim report, but doing so takes time. We will add responses if other issues come up. Thanks for listening."

Steve Walton Follow-up

AdoredTV Weighs In

Principled Technologies are the Publishers of BenchmarkXPRT; a series of benchmarks that are largely developed and funded by Intel and is also used to make performance claims by Intel.

The disclaimer is there as a requirement for Intel as a result of their previous meddling with BAPco which got them fined in excess of $1b by the FTC. The other requirement was that the FTC ordered that Intel claim it's stake in the benchmarking suite and outlet doing the testing whenever it makes a performance claim.

At this point it doesn't look very good for Intel. They've been caught commissioning a "third party" whom Intel has funded and had publish Intel's own BenchmarkXPRT series of tests to release benchmarks during an NDA period where they cannot be refuted, also under the guise of a third party.

Sources:

Principled Technologies Report

PCGamesN Reports on the Benchmarks

Steve Walton’s (TechSpot/HardwareUnboxed) Concerns

Steve Walton’s (TechSpot/HardwareUnboxed) Concerns - YouTube

TechSpot previous testing

Intel Statement

Gamers Nexus Rant

Gamers Nexus Gatecrash/Interview

Gamers Nexus Interview + Follow-up
Steve Walton Follow-up

AdoredTV's Take
Intel Benchmark Disclosure

Intel FTC Order

[Spotty]

This is why you always wait for independent reviews.

[D13H4RD]

Next on Avram Piltch's Hardware 

 

Just buy it: Why the Core i9 9900K is worth every penny 

 

When your whole life flashes before your eyes, how much of it do you want to not have 16 turbo threads? 

[ZeouLs]

[Shally]

Is 9th gen not going to be I7 9700k or have I got this all wrong

[NelizMastr]

6 minutes ago, KendoSapion said:

Is 9th gen not going to be I7 9700k or have I got this all wrong

The 9700K won't be the top dog, as it's going to be a 8c/8t. the i9 9900K will be the daddy, 8c/16t.

[leadeater]

7 minutes ago, KendoSapion said:

Is 9th gen not going to be I7 9700k or have I got this all wrong

It is but the content piece was focusing on the irregularities in the performance tests of known parts, the reason that matters is because the marketing is using performance comparisons against a competitor and those might not be correct or more accurately said 'Not reflective of actual performance or configuration of products'. 

[Shally]

1 minute ago, leadeater said:

It is but the content piece was focusing on the irregularities in the performance tests of known parts, the reason that matters is because the marketing is using performance comparisons against a competitor and those might not be correct or more accurately said 'Not reflective of actual performance or configuration of products'. 

Oh, so this is more of a "beware, last year the results were skewed by this much"? 

[leadeater]

2 minutes ago, KendoSapion said:

Oh, so this is more of a "beware, last year the results were skewed by this much"? 

Sort of, it's buyer beware but those are current test results carried out very recently and cross checked by Hardware Unboxed to check validity of performance shown in the Intel commissioned benchmark.

[Shally]

Just now, leadeater said:

Sort of, it's buyer beware but those are current test results carried out very recently and cross checked by Hardware Unboxed to check validity of performance shown in the Intel commissioned benchmark.

Ah okay makes more sense now, thanks for the help

[yian88]

Thats why you should buy AMD for now unless your business/life depends on it, if its just for desktop/gaming buy AMD, its like voting with your wallet.

Intel has been doing these bad practices since forever, i know you would say "but i buy whats best  bang for $ at the moment" which is not intel but those 5 extra FPS in gaming gets you to buy intel, yeah and then AMD makes no profits, cant compete, intel gets monopoly again, uses bad practices, sandbags performance and you get no better cpu's for 5 years+, locked down CPU's and motherboards that leave you with no upgrade path, also 1/3 of the die area of  silicon on these chips is useless iGPU, which you pay for.

I was surprised when i bought my Ryzen 2600 12 thread CPu its like a stock i7 8700 in performance for 1/3 of the cost, yes 1/3 because of low stocks all intel prices skyrocketed so now its the best time for AMD.

No fanboism just if you care at all about future hardware and prices (and if you are reading this article you should care) you should help AMD be competitive when they have good products, like Ryzen.

[Sauron]

16 minutes ago, KendoSapion said:

Ah okay makes more sense now, thanks for the help

More specifically, they said the 9900k is 50% faster than the 2700x but their 2700x tests are intentionally handicapped.

-edit-

also a 50% boost in framerate from the cpu alone sounds completely ridiculous, the only situation where you *might* see that is running 720p with sli 2080tis where the cpu is pushed to the limit...

[Arika]

you mean NDAs and Embargos that pretty much every company does these days?

 

if anyone buys any product based on paid reviews and benchmarks, they deserve to be burned

[mr moose]

So do we know if this journalist/publisher has got Intels testing legitimately or did they just score it and think "easy money"?

[ZestyJalapeno]

36 minutes ago, NelizMastr said:

The 9700K won't be the top dog, as it's going to be a 8c/8t. the i9 9900K will be the daddy, 8c/16t.

Wow. Even with actual competition Intel tries to get away with this level of piss banter. A top i7 K SKU with HT turned off so they can milk that cow? I don't care how many Ghz it says on the sticker. It's like they want people to buy a 2700X. Suck my dongle Intel.

[Trixanity]

I'd rather that they cherry pick benchmarks than post misleading (due to giving unfair conditions to competitors) benchmarks.

 

That's usually what AMD does as far as I'm aware. Cherry pick that is.

 

Alas, this is the usual scenario of "thank you AMD for making a competitive product so that I can buy a better/cheaper Intel/Nvidia product" (whenever applicable). In other words people are still going to be paying $488 for the Intel product instead of paying $329 for the AMD product in order to get those percentages and the brand.

[mr moose]

1 minute ago, Trixanity said:

I'd rather that they cherry pick benchmarks than post misleading (due to giving unfair conditions to competitors) benchmarks.

 

That's usually what AMD does as far as I'm aware. Cherry pick that is.

 

Alas, this is the usual scenario of "thank you AMD for making a competitive product so that I can buy a better/cheaper Intel/Nvidia product" (whenever applicable). In other words people are still going to be paying $488 for the Intel product instead of paying $329 for the AMD product in order to get those percentages and the brand.

Anyone would think the law of diminishing returns only applies to everything else.   Some people have the means to go further into the diminishing returns, that's up them no?

[Stefan Payne]

1 hour ago, Carclis said:

For me this news is quite concerning. 

Well, there are usually two sides to the story.

When such things are done, it, more often than not, means that the Hardware itself is shit and can't compete with the competition, thus you need something to cripple the competition and make you look better.

 

So that can mean that the 9900 might actually not be much better in games than the 8700k

[Carclis]

4 minutes ago, Trixanity said:

I'd rather that they cherry pick benchmarks than post misleading (due to giving unfair conditions to competitors) benchmarks.

 

That's usually what AMD does as far as I'm aware. Cherry pick that is.

 

Alas, this is the usual scenario of "thank you AMD for making a competitive product so that I can buy a better/cheaper Intel/Nvidia product" (whenever applicable). In other words people are still going to be paying $488 for the Intel product instead of paying $329 for the AMD product in order to get those percentages and the brand.

Cherry picking is what I expect to be honest. Paid benchmarks like these are usually to show off what the product does best. The problem is that conditions are not identical and extra measures are taken to deceive the consumer and ensure that the playing field is not even.

[Stefan Payne]

7 minutes ago, Trixanity said:

I'd rather that they cherry pick benchmarks than post misleading (due to giving unfair conditions to competitors) benchmarks.

Look on the bright side:
With stuff like this chances are pretty good that the difference between 8700K and 9900k are pretty small...

 

7 minutes ago, Trixanity said:

Alas, this is the usual scenario of "thank you AMD for making a competitive product so that I can buy a better/cheaper Intel/Nvidia product" (whenever applicable). In other words people are still going to be paying $488 for the Intel product instead of paying $329 for the AMD product in order to get those percentages and the brand.

Yeah and then "Oh, I would never buy AMD", next thing "Why isn't AMD competing with them right now??" 

Ähm, yeah, probably because you didn't buy AMD when it made sense and only used them to bring the prices down??

[mr moose]

1 minute ago, Carclis said:

Cherry picking is what I expect to be honest. Paid benchmarks like these are usually to show off what the product does best. The problem is that conditions are not identical and extra measures are taken to deceive the consumer and ensure that the playing field is not even.

yep, it's absolutely a dodgy comparison, Which makes me wonder if this PCgamesN is actually under an NDA.  It looks like they don't have the CPU to do a direct comparison on a proper bench.  Which makes me wonder if Intel actually gave them the tests to publish or if they acquired them by other means. 

[Carclis]

3 minutes ago, Stefan Payne said:

Well, there are usually two sides to the story.

When such things are done, it, more often than not, means that the Hardware itself is shit and can't compete with the competition, thus you need something to cripple the competition and make you look better.

 

So that can mean that the 9900 might actually not be much better in games than the 8700k

Well I don't think performance is the issue here or that they cannot compete. They just choose to command the same high prices because inevitably there will be enough people that just want the best regardless of the price premium.

[mr moose]

1 minute ago, Stefan Payne said:

Look on the bright side:
With stuff like this chances are pretty good that the difference between 8700K and 9900k are pretty small...

 

Why is it brighter to think that products are not getting better?  sometimes I really worry about the consumer mindset.

 

 

 

[Stefan Payne]

1 minute ago, Carclis said:

Well I don't think performance is the issue here or that they cannot compete.

Think you must not, know you must

 

1 minute ago, Carclis said:

They just choose to command the same high prices because inevitably there will be enough people that just want the best regardless of the price premium.

No, because they can, and there is no supply or there might be some slight issues with the i9-9900K. Seen that in the past that the prices were insanely high because of low yields

[Trixanity]

8 minutes ago, mr moose said:

Anyone would think the law of diminishing returns only applies to everything else.   Some people have the means to go further into the diminishing returns, that's up them no?

Of course it's your prerogative to pay for the diminishing returns. Diminishing returns is the name of the game.

 

In this case I just used this particular scenario as the example. There is no doubt in my mind that this should be faster in most or all workloads (higher frequency, equal thread count and software generally favoring Intel processors for multiple reasons) so it would be stupid to claim otherwise. However in the big picture I was referring to the same happening in scenarios being more equal or sometimes even favoring AMD. Branding is very powerful. Same goes for track records, purchasing habits and many other areas.