Intel 9th Gen Paid Benchmarks Take Advantage of NDA Periods

[Curufinwe_wins]

11 minutes ago, leadeater said:

I'm not sure what you read but there were multiple games far, very far, above margin of error difference in the PT updated document. I have gone and read a more trusted source to see what they got because well, game mode on for Ryzen is that stupid and by now I just don't actually care heh.

By the statement, I mean there was no situation in which Ryzen 3/5/7 was dramatically helped by turning on game mode. TR showed some cases, but Ryzen only showed cases where it either didn't matter or hurt performance.

 

Anyways, on the rest... I think I can try to argue about representative vs through, but I think in general, it is a function of our individual value matrices, and we don't actually predict dramatically different realities along each path. Just which is better or worse. So it was a good read.

[LinusTech]

7 minutes ago, leadeater said:

I'm not sure what you read but there were multiple games far, very far, above margin of error difference in the PT updated document. I have gone and read a more trusted source to see what they got because well, game mode on for Ryzen is that stupid and by now I just don't actually care heh.

 

I know in a lot of cases the difference isn't huge but when it is it effects performance just as much as for TR without it on, and those wildly inaccurate and not realistic performance numbers were picked as headline numbers for marketing by Intel to show just how great their product was.

 

What actually got so many people angry about this was that this commissioned test became or was allowed to become public before other 3rd party independent reviews were and was being passed off as just that when that's not actually the case. I don't know exactly who is mostly to blame for that but PCGamersN should never have published an article on it like they did in it's original form.

 

I get that it should really be ignored but this isn't like all the other marketing fluff presented during product launch in how it was presented and how it was reported on. It is generally known that numbers on a direct launch presentation and accompanying documents but the way the PT report came out to the public was not all that similar hence the confusion. 

 

Had no one raised any objections to the test data you can bet more tech news sites would have followed suit and publish similar articles and would have resulted in all likelihood systemic misinformation about the products for a good number of days past actual independent reviews, that's one of the big problems with inaccurate information is it doesn't just go away once shown as that.

 

Like you I can appreciate how well documented their procedure was but unfortunately when it comes down to it, and what matters the most to the interest public, it was flawed. In this particular sphere I would actually take a more poorly documented report that is correct over a well documented incorrect one, how you show the poorly documented one as correct is harder of course and that is just not something generally appreciated by the general gamer.

The really mind-blowing thing to me is that this ever made it as far as it did. Everyone I know at Intel is extremely cautious about mentioning competitors, let alone comparing to them... 

 

And these results are so obviously incorrect that it's baffling that no one saw them at nay point and went "whoa whoa whoa!" 

 

I dunno. 

 

Meanwhile at LTT we are already testing the 9900k so stay tuned for becmhmarks you can trust courtesy of Anthony 

[Curufinwe_wins]

1 minute ago, LinusTech said:

The really mind-blowing thing to me is that this ever made it as far as it did. Everyone I know at Intel is extremely cautious about mentioning competitors, let alone comparing to them... 

 

And these results are so obviously incorrect that it's baffling that no one saw them at nay point and went "whoa whoa whoa!" 

 

I dunno. 

 

Meanwhile at LTT we are already testing the 9900k so stay tuned for becmhmarks you can trust courtesy of Anthony 

Woah... woah... woah... I know you said you liked me Linus, but it's too early to consider hiring and trusting me to do this sort of thing. I don't even have a Canadian work permit.

 

 

Until Oct 19th!

[leadeater]

13 minutes ago, Curufinwe_wins said:

By the statement, I mean there was no situation in which Ryzen 3/5/7 was dramatically helped by turning on game mode. TR showed some cases, but Ryzen only showed cases where it either didn't matter or hurt performance.

Yea I didn't understand that properly, when you miss the key word being positive, oh well lol.

 

At least PT was able to demonstrate why you would pay them for such a report though, the deliverables they offer is indeed worth paying for and very well done. Funny thing is now that their name has been shoved in my face much more so because of this I am now remembering a lot of cases and reports I have read over the years that have been done by PT. They do indeed do a lot, or at least a lot of server and storage stuff that I have ended up reading in one way or another, getting major flashbacks of their company logo.

[Curufinwe_wins]

6 minutes ago, leadeater said:

Yea I didn't understand that properly, when you miss the key word being positive, oh well lol.

 

At least PT was able to demonstrate why you would pay them for such a report though, the deliverables they offer is indeed worth paying for and very well done. Funny thing is now that their name has been shoved in my face much more so because of this I am now remembering a lot of cases and reports I have read over the years that have been done by PT. They do indeed do a lot, or at least a lot of server and storage stuff that I have ended up reading in one way or another, getting major flashbacks of their company logo.

Heh one work of the too many walls of text I wrote tonight, so I couldn't assign blame if I wanted to.

 

Interesting! I certainly do not remember hearing about them before. From the details I did manage to gather, they seemed to be more of a broad company making occasional forays into all these different pots than consistently following and tracking all of the information actively (which admittedly would be a massive amount more work).

[leadeater]

6 minutes ago, Curufinwe_wins said:

Interesting! I certainly do not remember hearing about them before. From the details I did manage to gather, they seemed to be more of a broad company making occasional forays into all these different pots than consistently following and tracking all of the information actively (which admittedly would be a massive amount more work).

I had a look at their portfolio on their website, https://www.principledtechnologies.com/portfolio-marketing.php, they have done HEAPS! That's when it hit me how many times I've read something of theirs because there was stuff on there I had read or seen.

 

Edit:

I would note game testing is really not the realm of expertise.

[xriqn]

55 minutes ago, LinusTech said:

The really mind-blowing thing to me is that this ever made it as far as it did. Everyone I know at Intel is extremely cautious about mentioning competitors, let alone comparing to them... 

 

And these results are so obviously incorrect that it's baffling that no one saw them at nay point and went "whoa whoa whoa!" 

 

I dunno. 

 

Meanwhile at LTT we are already testing the 9900k so stay tuned for becmhmarks you can trust courtesy of Anthony 

I look forward to seeing those benchmarks soon :). I have no doubt in my mind that we'll see some pretty cool experiment videos with the 9900k after the benchmarking stuff too.

[Carclis]

14 minutes ago, xriqn said:

I look forward to seeing those benchmarks soon :). I have no doubt in my mind that we'll see some pretty cool experiment videos with the 9900k after the benchmarking stuff too.

#RIPGN

[xriqn]

1 minute ago, Carclis said:

#RIPGN

Haha yes I would love to see that happen with 9th gen intels too. The RTX competition was just pure entertainment.

[Stefan Payne]

On 10/11/2018 at 12:59 PM, mrthuvi said:

Even without benchmark, we can get a fairly educated guess of what the performance's gonna be like. 30% faster in multi-threaded workload compared to 8700k. Marginally better or tied with 8700k in gaming. It's for people who want BOTH single and multi-threaded performance AND willing to pay the hefty price premium (partially due to intel's supply problem). It's not some new magic tech, it's the same architecture since the 3-year-old skylake, just with moar cores. 

No, we do not know the scaling as it is never perfect and the 30% is only the theoretical increase due to increased core counts. You (almost) never see that in real life.

We do not  know if the internal connection for the cores is as efficient as it is with the previos ones.

We do not know how much the memory bandwith is limiting.

We do not know if there are other things that might (or might not) limit the performance.

 

Its just too soon to take even educated guesses as the architecture is not known, do they have a crossbar for the cores? Do they have a Ringbus? Or do they do the same thing that AMD did with Ryzen and make a cluster of 4 cores? 

 

And the most important thing:
How much did the power consumption change? How much can we expect with commercial available, useful software??

 

Because Amdahl's Law is a bitch and its better to wait and see for yourself when the product is finished and not speculate, even if the architecture seems to be the same. Because the performance of the Uncore is equally as important as the cores itself...

While it is highly likely that they still might be using the Ringbus but that also might increase the latency of that between the furthest cores, so loose performance.

 

And that is what you've also seen in the PT Documents, that the 8700K can be faster in some high framerate situations because of this higher latency.

[Carclis]

5 minutes ago, Stefan Payne said:

And that is what you've also seen in the PT Documents, that the 8700K can be faster in some high framerate situations because of this higher latency.

I think it's fair to say that there are enough issues with their testing that it's probably better to wait for Gamers Nexus or Anandtech before making that call.

[Stefan Payne]

7 minutes ago, Carclis said:

I think it's fair to say that there are enough issues with their testing that it's probably better to wait for Gamers Nexus or Anandtech before making that call.

...that is what I was saying...

You were the one making predictions

[Curufinwe_wins]

8 minutes ago, Stefan Payne said:

No, we do not know the scaling as it is never perfect and the 30% is only the theoretical increase due to increased core counts. You (almost) never see that in real life.

I think I'd err on the side of "the 8700k can often be within margin of error". Personally. 

 

But let's rephrase this. We know this is based on coffee NOT skylake-X. So we wouldn't expect a mesh. With well parallel processes that are not memory bandwidth limited. MT gains of 25% conservative can be expected (ideal scaling is 33%, so 75% of ideal). We have no reason to expect ccx type systems as Intel is including the standard igpu found in coffee Lake and thus you'd have to split resources, disable tons of extra die space, and somehow deal with the memory controller issue while being drop in compatible both forwards and backwards with CL boards and cpus. It would be a fiendishly complex rework for no reason, when ring scales pretty well anyways based on past die experience.

[MoonSpot]

1 hour ago, LinusTech said:

The really mind-blowing thing to me is that this ever made it as far as it did. Everyone I know at Intel is extremely cautious about mentioning competitors, let alone comparing to them... 

 

And these results are so obviously incorrect that it's baffling that no one saw them at nay point and went "whoa whoa whoa!" 

 

I dunno. 

 

Meanwhile at LTT we are already testing the 9900k so stay tuned for becmhmarks you can trust courtesy of Anthony 

If we take some minor liberties with assumptions we can quickly conclude that intel did not only know the numbers weren't good, but didn't care.  Additionally we know they aren't stupid enough to believe, for even a moment, that they'd get away with it with the tech community.

So what this means is that someone at intel did a cost benefit analysis and discovered that we, and their integrity, aren't worth a damn thing.

 

Conversely, imagine what it means if the people at intel saw these numbers and actually didn't realize they were bad(as per their claims of PTs #'s being inline with their findings).  These are people who are supposed to be engineers, or at least kinda smart.  But here we are with them not only being oblivious to the basics of the CPU space, but they've no feel for numbers either to the point where someone can say "This is 13600% more better here" and not bat an eye and be all like "yea bro, toe-da-lee legit man!  Let me hit that bong again after I snort these lines".

[leadeater]

48 minutes ago, Curufinwe_wins said:

when ring scales pretty well anyways based on past die experience.

Currently Ring is better for smaller, low core, dies anyway. Going Mesh would be a regression in performance for all current workloads and unnecessary, Ring works exceedingly well up to, very roughly, 16 cores. And that's 16 high end cores with more advanced targeted AVX2 workloads with quad channel memory, so I would think Ring will stick around on desktop until 12 cores.

 

Edit:

Pretty much we know how it will scale, we already have multiple products from Intel on Ring based architecture between 6 to 22/24 cores. 

[toor]

Found it kind of odd how this story was covered on TWS yesterday.  I get not wanting to burn bridges with Intel but to downplay it as everyone "freaking out and look other companies do it", was quite strange.   We SHOULD be 'freaking out' whenever this happens.   While techy people will spend a bit more time looking into them, I can assure you "executives", will do no such thing.  Those results WILL be shown in board meetings and taken as fact.

 

Best part will be if indeed despite running proper testing, Intel still comes out ahead because the problem is not in the results, it's how they tested.   I'll repeat what I said earylier in this thread -- Was Intel looking for a fallguy for if their sales were again low could they could try to pin it on them?   The whole thing stinks.

 

The lesson here to Intel and any other company (or blogger/tech channel) is simple - Truth is your bond.  History of Trust is paramount.   Intel has very little of it right dow for many reasons including Spectre / Meltdown.  Many of us who grew up on Ziff Davis publications no longer trust those companies to provide much of anything but paid promos disgused as reviews.

 

And yes, some of us still remember the "testing" done on Ryzen / Threadripper by people who have no idea what NUMA was let alone memory latency.  Yet we still have CPU benchmarks running at 4k+ (it's a bit invalid as it stresses the GPU more).

 

We almost need a wiki or something of industry lies.

[Droidbot]

2 hours ago, Stefan Payne said:

No, we do not know the scaling as it is never perfect and the 30% is only the theoretical increase due to increased core counts. You (almost) never see that in real life.

We do not  know if the internal connection for the cores is as efficient as it is with the previos ones.

We do not know how much the memory bandwith is limiting.

We do not know if there are other things that might (or might not) limit the performance.

 

Its just too soon to take even educated guesses as the architecture is not known, do they have a crossbar for the cores? Do they have a Ringbus? Or do they do the same thing that AMD did with Ryzen and make a cluster of 4 cores? 

 

And the most important thing:
How much did the power consumption change? How much can we expect with commercial available, useful software??

 

Because Amdahl's Law is a bitch and its better to wait and see for yourself when the product is finished and not speculate, even if the architecture seems to be the same. Because the performance of the Uncore is equally as important as the cores itself...

While it is highly likely that they still might be using the Ringbus but that also might increase the latency of that between the furthest cores, so loose performance.

 

And that is what you've also seen in the PT Documents, that the 8700K can be faster in some high framerate situations because of this higher latency.

It's 100% just 6-core CFL with two extra cores. 

 

With Skylake > Kaby Lake > Coffee Lake they swapped the microcode ID numbers (Skylake was 506E3, Kaby ES was 506E8, Kaby Final was 906E9), with 9th gen it utilises 906EC microcode (8th gen hexa is 906EA and quad is 906EB) -- this utilises the 8th gen microcode not new microcode. 

 

Also Splave (high end OCer, partnered with AsRock) has gotten 9th gen octa working on Z170M OC Formula with short+isolate method for hexa. 

 

It's traditional ringbus with same optimisations as 8th gen. 

[ZacoAttaco]

I found it interesting on the WAN Show today, Linus clearly has enough industry experience to know that manufacturer will 'fudge the numbers'. To the point where he was recounting times when companies like AMD and NVIDIA did it.

 

So for me, someone relatively new to tech hardware, I found this PT benchmark fiasco to be really unusual but it seems like it's pretty standard in the industry, people just know not to trust official benchmarks from the manufacturers themselves and to always wait for third-party reviews and cross-check them. I always learn so much from this channel and this community.

[MMKing]

35 minutes ago, ZacoAttaco said:

I found it interesting on the WAN Show today, Linus clearly has enough industry experience to know that manufacturer will 'fudge the numbers'. To the point where he was recounting times when companies like AMD and NVIDIA did it.

 

So for me, someone relatively new to tech hardware, I found this PT benchmark fiasco to be really unusual but it seems like it's pretty standard in the industry, people just know not to trust official benchmarks from the manufacturers themselves and to always wait for third-party reviews and cross-check them. I always learn so much from this channel and this community.

This particular example is rather tame as far as subtlety is concerned. It seems just a select few ''journalists'' actually believed these numbers. There are far more elaborate tricks being used.

 

In game benchmark suites tailored for certain CPUs and/or GPUs. Not just in favor of either AMD, Nvidia or Intel. But for example, benchmark runs tailored to favor Turing, while targeting Pascal's weak points to make the difference between Turing and Pascal seem greater.

 

Pascal's compression does not work well with 10 bit color at all. Naturally, all of Turing's Nvidia slides showcasing the performance difference is done at 10 bit color.

 

The only reason the 2700x vs 9900k paid benchmark is run at 1080p, is because after testing at numerous resolutions and settings. They found that 1080p best showcase the difference between the 9900k and the 2700x. Nevermind the fact that Gears of War 4 runs at over 300 frames per second. The Warhammer 2 laboratory test is explicitly stated to be ''powered by Intel''. Counter Strike: Go runs at 300FPS. Forza 7 runs at over 180 FPS. Rainbow six siege runs at over 280FPS. PUBG over 200FPS. GTA 5 over 170FPS.

 

I mean, the discussion so far has been on the ''game mode'' farce. Memory discussions and the choice to use an aftermarket cooler on every CPU but the 2700x, a cooler which is not designed for the TR4 platform even. But there has been little mention that none of these settings represent actual real world scenarios. No one is gonna buy a 1080 + 580USD CPU, then run their games 1080p at hundreds of frames per second except for maybe Counter Strike: GO. Getting as many % difference between the 9900K and the 2700x was the ONLY important factor.

 

[Misanthrope]

1 hour ago, toor said:

Found it kind of odd how this story was covered on TWS yesterday.  I get not wanting to burn bridges with Intel but to downplay it as everyone "freaking out and look other companies do it", was quite strange.   We SHOULD be 'freaking out' whenever this happens.   While techy people will spend a bit more time looking into them, I can assure you "executives", will do no such thing.  Those results WILL be shown in board meetings and taken as fact.

 

Best part will be if indeed despite running proper testing, Intel still comes out ahead because the problem is not in the results, it's how they tested.   I'll repeat what I said earylier in this thread -- Was Intel looking for a fallguy for if their sales were again low could they could try to pin it on them?   The whole thing stinks.

 

The lesson here to Intel and any other company (or blogger/tech channel) is simple - Truth is your bond.  History of Trust is paramount.   Intel has very little of it right dow for many reasons including Spectre / Meltdown.  Many of us who grew up on Ziff Davis publications no longer trust those companies to provide much of anything but paid promos disgused as reviews.

 

And yes, some of us still remember the "testing" done on Ryzen / Threadripper by people who have no idea what NUMA was let alone memory latency.  Yet we still have CPU benchmarks running at 4k+ (it's a bit invalid as it stresses the GPU more).

 

We almost need a wiki or something of industry lies.

Agreed: If Linus is truly so jaded and cold about this issues that it seems normal and not worth talking about as he lead to believe with that segment he shouldn't be covering news.

 

Can't help but think he's more concerned with all the free xeons he gets if he thinks people will give intel a pass with this 'But everyone does it!' Bullshit take.

[toor]

Hard core gamer, Enthusiast, call the segment whatever you want.  We are discussing it and well aware of what tests are not real world.   Those aren't the target markets that are impacted most by this kind of deception however. 

 

Yet I'm still amazed at the level of variance in hardware that is considered acceptable.   If there's one thing PI had right (at least their claim later) apart from other testing it was to target real world results and less on breaking records.  What I mean is if review sites like LMG are stockpiling and using hardware they know represents even 10% of the best yields ("God Tier"), those tests are complete GARBAGE to BOTH segments.   Much (if any?) of their testing is not about breaking OC benchmarks.  There is a difference between ordering hardware from an OEM  and VAR.  The latter will do that certification (Alienware, IBP, etc).

 

Does LMG aquire hardware through another white label distributor or are they direct?

[Carclis]

2 hours ago, MMKing said:

This particular example is rather tame as far as subtlety is concerned. It seems just a select few ''journalists'' actually believed these numbers. There are far more elaborate tricks being used.

Well I think that's the problem. The stance that seemed to have been taken is "well other companies have done worse so nice try Intel" instead of any sort of condemnation. Apparently there was no malice or ill intent despite Intel stating that they stand by and that their own laboratory testing mirrors the results of PT, which benchmarked "real world" gaming setups. I dunno about you but I don't know anybody who would consider the amounts of RAM they used, graphical settings (for the hardware setup) or game mode on a Ryzen 2700x a "real world" configuration. I don't see how that isn't deceptive on Intel's part.

[Falkentyne]

On 10/11/2018 at 7:57 AM, Syntaxvgm said:

Why did you have to post this?
This just made me depressed again almost to the point of suicide.  Literally all the negative things in gaming (besides the newsgroup space sim / flight sim drama) just all together.  Especially the Ultima posts.  Thank you.

 

Should have left what EA did to Interplay and Bard's Tale (back when Dragon Wars was supposed to be Bard's Tale IV).

 

Something to make you happy:

Back when EA was run by GAMERS, they wanted to fully embrace the Commodore Amiga ....

[D13H4RD]

Kind of begs the question of how is this allowed to happen though 

 

57.4% is a massive performance delta. That should’ve set an immediate red flag if anyone at Intel actually did do proper internal testing.

[mr moose]

6 minutes ago, D13H4RD2L1V3 said:

Kind of begs the question of how is this allowed to happen though 

 

57.4% is a massive performance delta. That should’ve set an immediate red flag if anyone at Intel actually did do proper internal testing.

 Because even though you and I and half the world knows there is a difference between their tests and what the average consumer is going to experience, their tests are still accurate as detailed, there is no room for bad results in business, if you don't have good results you make the average results look good.   It is literally suicide for any company making any product not to give their wares its absolute best chance of selling no matter how close to a lie their claims maybe.

 

Just be grateful the PC world can have independent benchmarks readily available, imagine if it was like the supplements industry where the ads leave you with the distinct impression they can cure cancer, depression, old age and make you attractive and immune to hangovers.