Passmark - AMD "Decimated" In New Update

[Lurick]

It would seem that someone is up to some shenanigans with PassMark this time following not too far behind UserBenchmark's recent screwings around with weighting. This time however PassMark has claimed to be "transparent" in their implementation however this seems to be rather suspect as the highest ranked AMD CPU on the new v10 list is 35th on the list and it's an iGPU variant, the 4300U, which hasn't even been tested on the v10 benchmark, but somehow beats out. There is talk that they have introduced an AVX-512 portion or subset to the benchmark which AMD doesn't even have instructions for and will almost certainly give favor to Intel CPUs.

 

Quote:

Quote

PassMark has implemented a new build of its PerformanceTest that has left more than a few eyebrows being raised. While the PerformanceTest V9 CPU list from March 4 showed AMD processors hogging all the high positions apart from a few Intel intruders, the V10 list has seen Team Red virtually wiped out of contention. The top 34 places are all occupied by chips from Team Blue, while the AMD Ryzen 3 4300U is the only rival part in the top 40.

 

Thoughts:

I would seriously love to see how Passmark claims this to be fair and how their CPU benchmark can put the 4300U a 2.7GHz base and 3.7GHz boost CPU ABOVE other Ryzen 3000 series CPUs which have much higher base and boost clocks from the start. It would seem that this is just another PR stunt to try and help Intel (for some reason or another) that will come out as an "accident" once enough people start digging into it.

 

Source:

https://www.notebookcheck.net/AMD-obliterated-from-PassMark-s-single-thread-performance-chart-as-Intel-claims-top-34-places-thanks-to-version-10-update.457506.0.html

 

V9 :https://www.cpubenchmark.net/pt9_cpu_list.php

V10: https://www.cpubenchmark.net/singleThread.html

[Sauron]

Damn. Well, if someone ever posts a passmark result just refer them to cinebench.

[Neftex]

how can someone smart enough to develop that kind of software fuck up so badly to misrepresent the data it collects. they cant be this stupid right? its paid off

[ARikozuM]

I can understand using math to change a 16-core to a 4-core score, but this? This is definitely for ad money. 

[Sauron]

6 minutes ago, Neftex said:

how can someone smart enough to develop that kind of software fuck up so badly to misrepresent the data it collects. they cant be this stupid right? its paid off

Cumulative benchmarks are a pretty stupid idea anyway, they don't really give you any useful information for any specific workload because the results are all mashed up.

[porina]

47 minutes ago, Lurick said:

It would seem that someone is up to some shenanigans with PassMark this time following not too far behind UserBenchmark's recent screwings around with weighting.

I'm digging into what they're testing. At a very high level overview, for non-heavily CPU optimised code we'd expect Zen 2 to have an advantage in IPC, offset by typically lower clocks. If as they say they are optimising more for modern architectures, that could get interesting where more specialised instructions are concerned. 

 

Note there is a also a multi-thread ranking on their site, lead by the new threadrippers first, some Epycs in there and the 3950X before you even reach the 1st Intel. So the question is only about if the single thread rank is meaningful or not.

https://www.cpubenchmark.net/high_end_cpus.html

 

This almost feels like stoking outrage just because attack Intel.

[Curufinwe_wins]

13 minutes ago, porina said:

I'm digging into what they're testing. At a very high level overview, for non-heavily CPU optimised code we'd expect Zen 2 to have an advantage in IPC, offset by typically lower clocks. If as they say they are optimising more for modern architectures, that could get interesting where more specialised instructions are concerned. 

 

Note there is a also a multi-thread ranking on their site, lead by the new threadrippers first, some Epycs in there and the 3950X before you even reach the 1st Intel. So the question is only about if the single thread rank is meaningful or not.

https://www.cpubenchmark.net/high_end_cpus.html

 

This almost feels like stoking outrage just because attack Intel.

Also avx is actually a huge deal for code that can use it. 

 

But with that said, the single threaded stuff does still look really odd.

[porina]

1 minute ago, Curufinwe_wins said:

Also avx is actually a huge deal for code that can use it. 

Zen 2 is roughly at parity with Intel consumer in AVX2 throughput. AVX-512 is where Intel could take the lead, depending on if the CPU has it at all, if it has the 1 or 2 unit implementation, and if software uses it. Basically Zen 2 shouldn't fall behind on software that uses it. Zen 1 is really weak in that area though.

[Unixsystem]

3 minutes ago, porina said:

Zen 2 is roughly at parity with Intel consumer in AVX2 throughput. AVX-512 is where Intel could take the lead, depending on if the CPU has it at all, if it has the 1 or 2 unit implementation, and if software uses it. Basically Zen 2 shouldn't fall behind on software that uses it. Zen 1 is really weak in that area though.

To my knowledge AVX-512 is only on the Intel HEDT parts, but the consumer SKUs are at the top of the new Passmark list. Really seems like they are pulling some real obscure optimizations out of their ass to get Intel on top. 

[Lurick]

10 minutes ago, porina said:

Zen 2 is roughly at parity with Intel consumer in AVX2 throughput. AVX-512 is where Intel could take the lead, depending on if the CPU has it at all, if it has the 1 or 2 unit implementation, and if software uses it. Basically Zen 2 shouldn't fall behind on software that uses it. Zen 1 is really weak in that area though.

I would say if you're going to use specific AVX instruction sets that should be called out pretty early on that could favor one vendor over the other, I don't care who it favors just say "Hey this is using xyz instruction which is geared towards abc and as such vendor blah could see higher performance" or "Vendor so-and-so doesn't implement instructions we use for now so until then their products might appear lower on the chart.", something to call out the discrepancy and why it changed. However, I think what's really off about this is not only that a 4300U is the top AMD CPU but there are tons of CPUs on the list from both sides that have one sample only and no way, that I can see, to filter out stuff with high chances for anomaly. This of course goes for the entire site, not just the single threaded benchmark.

[porina]

17 minutes ago, Unixsystem said:

To my knowledge AVX-512 is only on the Intel HEDT parts, but the consumer SKUs are at the top of the new Passmark list. Really seems like they are pulling some real obscure optimizations out of their ass to get Intel on top. 

I'm still filling up a table by hand, which is pretty tedious and time consuming. For now it is looking like (for desktop CPUs at least) the rankings fall more or less in line with the CPU max turbo clock on both sides. I should have a clearer picture soon, but it doesn't look like AVX-512 HEDT parts are doing any better than their similarly clocked mainstream parts.

 

I need to decode the mobile part numbers, as some have AVX-512, some don't. See how they fit in then.

 

13 minutes ago, Lurick said:

I would say if you're going to use specific AVX instruction sets that should be called out pretty early on that could favor one vendor over the other, I don't care who it favors just say "Hey this is using xyz instruction which is geared towards abc and as such vendor blah could see higher performance" or "Vendor so-and-so doesn't implement instructions we use for now so until then their products might appear lower on the chart.", something to call out the discrepancy and why it changed. However, I think what's really off about this is not only that a 4300U is the top AMD CPU but there are tons of CPUs on the list from both sides that have one sample only and no way, that I can see, to filter out stuff with high chances for anomaly. This of course goes for the entire site, not just the single threaded benchmark.

AVX2 support has been around Haswell on Intel side, and I'm not sure exactly which AMD CPU but it is in the 'dozer/'driver era. Before Ryzen at any rate. If it makes sense to use it, software should use it. I don't feel AVX2 needs to be called out specifically as it is long established, in a similar way we don't break down every instruction family in common use.

 

For AVX-512, I might start to agree in part, as that is still relatively new.

 

While I'm not familiar with PasMark, is this another one of those average user submission sites? The 4300U looks like an outler and should be disregarded. The 2nd highest AMD is a 3900 non-X which also doesn't make much sense unless it had one user that overclocked the hell out of it. The highest one then becomes 3950X and it does seem to be hanging around some slightly lower clocked Intel CPUs for what its worth. 

[Curufinwe_wins]

1 hour ago, porina said:

Zen 2 is roughly at parity with Intel consumer in AVX2 throughput. AVX-512 is where Intel could take the lead, depending on if the CPU has it at all, if it has the 1 or 2 unit implementation, and if software uses it. Basically Zen 2 shouldn't fall behind on software that uses it. Zen 1 is really weak in that area though.

With AVX2 even Zen2 is behind. Not by an insurmountable amount, but to the extent that the 8C ~5Ghz chip matches the 12C ~4.3Ghz from AMD heh

[porina]

7 minutes ago, Curufinwe_wins said:

With AVX2 even Zen2 is behind. Not by an insurmountable amount, but to the extent that the 8C ~5Ghz chip matches the 12C ~4.3Ghz from AMD heh

Got some example software that shows that behaviour? Maybe I've been skewed by my own interests, but Zen 2 seems to be "a bit" ahead in peak IPC compared to Intel mainstream when it comes to FMA workloads. Assuming FMA is part of AVX2? This is like a quiz show now. The host asks, am I sure? I'm now not sure of anything at all...

[Curufinwe_wins]

21 minutes ago, porina said:

Got some example software that shows that behaviour? Maybe I've been skewed by my own interests, but Zen 2 seems to be "a bit" ahead in peak IPC compared to Intel mainstream when it comes to FMA workloads. Assuming FMA is part of AVX2? This is like a quiz show now. The host asks, am I sure? I'm now not sure of anything at all...

The HPC Linpack Benchmark I showed previously that you originally quoted. The 9900k doesn't support AVX512. Bottom two scores. Also see how the 32C older Zen chip struggles so much by comparison (given its thread count).  Also see how the 9800X which does support AVX512 storms way ahead even at lower frequency than the 9900k.

[porina]

19 minutes ago, Curufinwe_wins said:

The HPC Linpack Benchmark I showed previously that you originally quoted. The 9900k doesn't support AVX512. Bottom two scores. Also see how the 32C older Zen chip struggles so much by comparison (given its theoretical throughput ofc). 

Oh, that pic wasn't there when I hit quote! I'm not too familiar with the details behind linpack but it certainly is no Cinebench. Bandwidth matters. How, I'm not sure.

 

I'm not familiar with the Xeons, but the 9800X we know is monolithic. With a 2 unit AVX-512 implementation, peak AVX-512 throughput (IPC) is double AVX2 in FP64 workloads like this. The 2990WX is not monolithic, you have the horribleness of NUMA nodes and varying distances back to ram. I suspect that is choking it badly, combined with the roughly half of Intel FP64 IPC in that generation.

 

Similar might apply to the 9900K, with its unified cache and monolithic construction. The 3900X might not have NUMA nodes any more, but it is still far from monlithic. It is best seen logically as 4 CCX of 3 cores each, since if a task has to leave a CCX you start to hit additional latency and bandwidth usage. On consumer models they went with half bandwidth writes from each chiplet which also wont help. I never got a clear answer if the server versions were full bandwidth or not.

 

Earlier when I was talking about peak AVX2 (specifically FP64) performance, Zen 2 does seem ahead. By peak I mean, if other limits don't get in the way. I'm trying to help out on another forum where some users of the 3950X and 3970X have reported lower than expected performance in a Prime95-like scenario my understanding says should not be impacted. The only clue I have to go on is that the multiple chiplet implementation is showing limits not previously seen before.

[Curufinwe_wins]

1 minute ago, porina said:

Oh, that pic wasn't there when I hit quote! I'm not too familiar with the details behind linpack but it certainly is no Cinebench. Bandwidth matters. How, I'm not sure.

 

I'm not familiar with the Xeons, but the 9800X we know is monolithic. With a 2 unit AVX-512 implementation, peak AVX-512 throughput (IPC) is double AVX2 in FP64 workloads like this. The 2990WX is not monolithic, you have the horribleness of NUMA nodes and varying distances back to ram. I suspect that is choking it badly, combined with the roughly half of Intel FP64 IPC in that generation.

 

Similar might apply to the 9900K, with its unified cache and monolithic construction. The 3900X might not have NUMA nodes any more, but it is still far from monlithic. It is best seen logically as 4 CCX of 3 cores each, since if a task has to leave a CCX you start to hit additional latency and bandwidth usage. On consumer models they went with half bandwidth writes from each chiplet which also wont help. I never got a clear answer if the server versions were full bandwidth or not.

 

Earlier when I was talking about peak AVX2 (specifically FP64) performance, Zen 2 does seem ahead. By peak I mean, if other limits don't get in the way. I'm trying to help out on another forum where some users of the 3950X and 3970X have reported lower than expected performance in a Prime95-like scenario my understanding says should not be impacted. The only clue I have to go on is that the multiple chiplet implementation is showing limits not previously seen before.

That's all fair. I'd have to pull up the 3700x, but I will say that Linpack is one of the most highly optimized codes in the world and was originally developed for supercomputer testing, so I doubt the NUMA issue is a big one (bandwidth however is possible, even if I don't see as particularly likely). That was from a Puget Systems article a while back.

[RejZoR]

Isn't this the second time they are pulling this shit? Anyone still takes this company seriously? They are so obviously pandering to Intel even Ray Charles can see it... And he's blind and also dead.

[porina]

3 minutes ago, Curufinwe_wins said:

That's all fair. I'd have to pull up the 3700x, but I will say that Linpack is one of the most highly optimized codes in the world and was originally developed for supercomputer testing, so I doubt the NUMA issue is a big one (bandwidth however is possible, even if I don't see as particularly likely). That was from a Puget Systems article a while back.

That's a good point. Linpack is used for the Top500 supercomputer listing that is published twice a year I think. However they don't simply just download it and push run. They have to optimise it to the characteristic of their particular system. I don't know if generic downloadable versions of Linpack for Windows will have the same level of system-specific optimisation behind it.

 

Personally I'd like to see Y-cruncher used more for testing. It has been used to set at least the last two digits of Pi record computations. The writer of it has spent time optimising it for many architectures, most recently a Zen 2 update giving ~10% ball park uplift from the best previous version. It can use AVX-512 also, which is still stressful although not as much as Prime95. Again it isn't Cinebench, ram bandwidth matters to a degree.

[porina]

Ok, I've done a table of selected CPUs and got the following:

 

PassMark v10 ST rank	CPU	Max turbo	Arch
1	Intel Core i9-9900KS @ 4.00GHz	5
2	Intel Core i9-9900K @ 3.60GHz	5
3	Intel Core i9-9900KF @ 3.60GHz	5
5	Intel Core i7-9700K @ 3.60GHz	4.9
6	Intel Core i7-9700KF @ 3.60GHz	4.9
7	Intel Core i9-9900 @ 3.10GHz	5
9	Intel Core i7-8086K @ 4.00GHz	5
10	Intel Core i7-9700F @ 3.00GHz	4.7
11	Intel Core i7-9700 @ 3.00GHz	4.7
14	Intel Core i7-10750H @ 2.60GHz		Comet?
18	Intel Core i5-9600K @ 3.70GHz	4.6
19	Intel Core i7-8700K @ 3.70GHz	4.7
21	Intel Core i5-9600 @ 3.10GHz	4.6
24	Intel Core i5-9600KF @ 3.70GHz	4.6
27	Intel Core i7-7700K @ 4.20GHz	4.5
30	Intel Core i7-8700 @ 3.20GHz	4.6
31	Intel Core i9-10940X @ 3.30GHz	4.6
33	Intel Core i9-10900X @ 3.70GHz	4.5
34	Intel Core i9-10980XE @ 3.00GHz	4.6
35	Intel Core i7-7740X @ 4.30GHz	4.5
36	AMD Ryzen 3 4300U	3.7
37	Intel Core i9-10920X @ 3.50GHz	4.6
40	Intel Core i5-9500 @ 3.00GHz	4.4
43	Intel Core i3-9350KF @ 4.00GHz	4.6
44	Intel Core i5-10300H @ 2.50GHz		Comet?
45	AMD Ryzen 9 3900	4.3
48	Intel Core i5-9500F @ 3.00GHz	4.4
51	Intel Core i7-9800X @ 3.80GHz	4.4
54	AMD Ryzen Threadripper 3990X	4.3
57	AMD Ryzen Threadripper 3960X	4.5
59	AMD Ryzen 9 3950X	4.7
60	AMD Ryzen 7 PRO 3700	4.4
62	Intel Core i5-8600K @ 3.60GHz	4.3
63	Intel Core i9-7900X @ 3.30GHz	4.3
66	Intel Core i3-9320 @ 3.70GHz	4.4
67	Intel Core i9-9900X @ 3.50GHz	4.4
68	AMD Ryzen 9 3900X	4.6
70	Intel Core i5-8600 @ 3.10GHz	4.3
72	Intel Core i3-9100F @ 3.60GHz	4.2
73	Intel Core i3-9300 @ 3.70GHz	4.3
75	AMD Ryzen 9 PRO 3900	4.3
76	Intel Core i7-6700K @ 4.00GHz	4.2
77	Intel Core i5-1035G7 @ 1.20GHz	3.7	Ice
78	AMD Ryzen 7 3800X	4.5
80	Intel Core i5-3170K @ 3.20GHz		???
81	AMD Ryzen Threadripper 3970X	4.5
82	Intel Core i7-10510U @ 1.80GHz	4.9	Comet
83	Intel Core i9-9920X @ 3.50GHz	4.4
84	Intel Core i5-1035G4 @ 1.10GHz	3.7	Ice
85	Intel Core i9-7920X @ 2.90GHz	4.3
86	Intel Core i9-7940X @ 3.10GHz	4.3
88	Intel Core i9-9960X @ 3.10GHz	4.4
89	AMD Ryzen 7 3700X	4.4
90	AMD Ryzen 5 3600X	4.4
91	Intel Core i3-9100 @ 3.60GHz	4.2
96	Intel Core i9-9980XE @ 3.00GHz	4.4
97	Intel Core i9-7980XE @ 2.60GHz	4.2
99	Intel Core i7-7700 @ 3.60GHz	4.2
100	Intel Core i5-9400F @ 2.90GHz	4.1
101	Intel Core i5-7640X @ 4.00GHz	4.2	Kaby
102	Intel Core i7-7820X @ 3.60GHz	4.3
103	Intel Core i9-9940X @ 3.30GHz	4.4
104	Intel Core i7-7900X @ 3.30GHz	4.3
105	Intel Core i9-9820X @ 3.30GHz	4.1
109	Intel Core i7-1065G7 @ 1.30GHz	3.9	Ice
110	Intel Core i7-10710U @ 1.10GHz	4.7	Comet
111	Intel Core i3-7350K @ 4.20GHz	4.2
114	Intel Core i3-8350K @ 4.00GHz	4
116	AMD Ryzen 5 PRO 3600	4.2
117	AMD Ryzen 5 3500X	4.1
119	Intel Core i5-8500 @ 3.00GHz	4.1
122	Intel Core i7-4790K @ 4.00GHz	4.4	Haswell
124	Intel Core i5-10210U @ 1.60GHz	4.2	Comet
126	Intel Core i7-7800X @ 3.50GHz	4
128	AMD Ryzen 5 3600	4.2

Next to each CPU I've also listed the max turbo clock, which might be hit running single thread tasks. I've also wrote in some architectures which may help to understand what's going on. I excluded Xeons, low power desktop CPUs, and older mobile CPUs. I only looked at rankings down to 128th position, which is where the Ryzen 3600 sits.

 

Intel mainstream desktop CPUs I put in mid-blue (and some unsorted ones). Light blue are Intel 10th gen mobile CPUs. Dark blue are Intel HEDT. Red is all AMD, dark red Threadripper.

 

As a rough eyeball looking only at Intel desktop and HEDT CPUs for starters, there seems to be a good trend faster turbo = higher up ranking. Of note, the HEDT CPUs are about parity with the mainstream ones. As such this test result doesn't seem to be significantly affected by having AVX-512.

 

10th gen mobile are split into Comet Lake and Ice Lake. Comet lake is still essentially Skylake and judging by the boost clock, it fits in nicely. Ice Lake is the new Sunny Cove architecture and it seems to have gained a bit in IPC, with lower specified clocks compared to CPUs around it. I couldn't find the H CPUs on Intel's site, but references elsewhere suggest they are 45W Comet Lake parts, so they're unlikely to be power constrained and have better cooling too.

 

The 4300U I think we have to ignore as an outlier. It isn't a common CPU, and as a user submitted result database someone with one overclocked could skew its position significantly. If we look at the wider picture, there isn't that much of a spread between all of these CPUs based on the scores on the site. The top ranked CPU has a score just under 25% higher than the 100th ranked CPU. If we look at the clocks, we go from around 4.1 to 5.0 (22%). It is in the general ball park.

 

How do the AMD CPUs slot in amongst the Intel ones? There is a bit of variation but it seems about parity. I actually would have expected some IPC advantage in AMDs favour actually, but there doesn't appear to be an obvious one. Again, as these are based on user submitted results, we can speculate on how good they might be compared to ideal benching situations. There are many possible factors that may skew the results. For example, I'd consider the bundled coolers with Zen 2 CPUs to be inadequate for serious computational workloads, and AMD throttle the CPU clock back with temperatures earlier than Intel does. Something like that may be a contributing factor.

[CTR640]

What even is AVX512 and which programs uses that? I don't buy any of this filthy bullshit from a filthy business named Intel. They are known paying off to make their cpu's look better and faster while in reality, they are nothing but a colander full of security bugs and other fuck-ups. "Oh intel cpu is faster in passmark!!!11!!one!1eleven!! so RiP pO0R aMD!!! Really Intel, go #@$% yourself.

[Taf the Ghost]

https://www.passmark.com/forum/pc-hardware-and-benchmarks/46757-single-thread-score-rating?p=46873#post46873

 

@porina @Lurick here's the Passmark dev discussing the changes and the effects. Notebookcheck is clickbaiting a bit. It really seems V9 was actually sandbagging Zen2 a bit and the V10 now makes the single-core test very sensitive to Clocks. (And it slams a bunch of the Intel Mobile parts.)

 

Given that Zen2 and Skylake have generally similar IPC (on net, per-tasks there's up to 25% differences; see The Stitl's in-depth testing), this actually isn't too surprising. But Passmark should probably clarify what the point of their test. Not that pure single-core really matters for anything.

[Taf the Ghost]

46 minutes ago, CTR640 said:

What even is AVX512 and which programs uses that? I don't buy any of this filthy bullshit from a filthy business named Intel. They are known paying off to make their cpu's look better and faster while in reality, they are nothing but a colander full of security bugs and other fuck-ups. "Oh intel cpu is faster in passmark!!!11!!one!1eleven!! so RiP pO0R aMD!!! Really Intel, go #@$% yourself.

AVX is 128bit vector operations; AVX2 is 256bit and AVX512 is, as you can guess, 512bit operations.  https://en.wikipedia.org/wiki/Advanced_Vector_Extensions  

 

They're also what kills Intel's CPUs when they try to use them, as the units produce a lot of heat to do a calculation. AMD's current response is "just use a GPU, you moron", because that's what GPUs do. They're used in very little at the moment because Intel only just launched any normal consumer parts with AVX512 units (Icelake Mobile). 

[xAcid9]

Is it not possible for big techtuber like LTT to make a video about these type of websites that misinformed general consumer? 

[descendency]

I hate synthetic benchmarks for this reason. 

[williamcll]

What if these are all single threaded gaming benchmarks.