Ryzen 5000 Series Reviews Discrepancy

[RacA]

Disclaimer: I am posting this in the tech news category as this is related to the recent Zen 3 Launch, Linus and Luke has nudged the topic in the most recent WAN show 06/11/20 but in my opinion, there is more to it.
 

Zen 3 has launched and you're after a new cpu you probably have seen at least several benchmarks by different tech youtubers and news outlets by now.
Some of them show new ryzens beating intel by quite a margin, and some of them put them on par with i9-10900k and ever slower intel SKUs.

This has been addressed by LTT in the latest WAN show episode.
Linus attributes the discrepancies in reviews to the RAM configurations used for testing, but there is one more factor at play here.
 

Uh... is our Ryzen 5000 Review WRONG?? - WAN Show November 6 , 2020

Some reviewers are still using rtx 2080 ti or using 3080 but testing on 1080p ultra settings. One would think that it should not matter much for 1080p testing, right? Not quite.
If we'll look at 1% and 0.1% low performance, on this chart below (Gamers Nexus 5600x review) we can spot that all new Ryzens are performing better than i9-10900k on average, yet Intel still has better lows:



This appears to be the case across many games, the reason Ryzen comes out on top is that above test is done on 1080p medium, not ultra.

Anandtech included results for various settings and resolutions which is a helpful example:
F1 2019 768p ultra low, new Ryzens stomp intel into the ground, same game on 1080p ultra and... how come it's so blue at the top of the chart?

Conclusion:
It occurs to me that new Ryzens have much higher performance ceiling than we thought, the reason why reviews end up being so different is the unexpected GPU bottleneck.
When new Zen 3 CPUs and Intel 10th gen are limited by a GPU on the top end, Intel comes out on top due to better low frame performance.
When there is no (or lesser) limit imposed by GPU, even ryzen 5 5600x beats i9-10900k in the average FPS. This is especially visible in CS:GO benchmarks which is one of the least GPU intensive titles.

It looks like low frame performance improves for Zen 3 CPUs by a mile, with a faster RAM kit.
That's why in LTT review, 1% low fps results look to be on par with Intel, as they have used 3600Mhz CL14 kit, contrary to multiple reviewers that are using 3200Mhz kits.
 

Sources

-Wan show
-Gamers Nexus 5600x review
-Anandtech review
-My own reflections after watching/reading bazillion reviews

[mbntr]

that doesn't make much sense, a slower GPU shouldn't be causing that, even with the 3080 intel has better lows.

I would say it's either the architecture, the ram or just bad optimization for the new architecture.

[CarlBar]

15 minutes ago, mbntr said:

that doesn't make much sense, a slower GPU shouldn't be causing that, even with the 3080 intel has better lows.

I would say it's either the architecture, the ram or just bad optimization for the new architecture.

 

Once the GPU bottlenecks it comes down to how 1% and 0.1% low performance impacts average framerate. Intel does better in that department though it appears to be heavily RAM influenced.

[leadeater]

1 minute ago, CarlBar said:

 

Once the GPU bottlenecks it comes down to how 1% and 0.1% low performance impacts average framerate. Intel does better in that department though it appears to be heavily RAM influenced.

I'm sure if there is enough noise GN Steve will do a bunch of tests across ram speeds and timing, and IF clocks, to see how that impacts things.

[Curufinwe_wins]

This is still interesting. Presuming there is no bug in the matter, it seems to suggest that there is a notable difference in the "smoothness" of performance being presented by Intel's current offerings.

 

Maybe this is nothing more or less than a symptom of being monolithic vs chiplet, where even though Ryzen is faster 99% of the time, some accesses to some core complexes can end up slower than than the performance delta between Intel and Zen 3? It might seem to make sense, but a convincing explanation doesn't make it right of course.  At more heavily GPU bound situations then, the tiny fluctuations that might occur now become accentuated compared to the whole as the rest of the time, framerate is tied to a uniform level.

[porina]

2 hours ago, RacA said:

It occurs to me that new Ryzens have much higher performance ceiling than we thought, the reason why reviews end up being so different is the unexpected GPU bottleneck.

You can't say the GPU "bottleneck" is unexpected. It is one reason reviewers have to include insanely low resolutions/settings that no actual user would use, just to show the difference the CPU makes.

 

2 hours ago, RacA said:

That's why in LTT review, 1% low fps results look to be on par with Intel, as they have used 3600Mhz CL14 kit, contrary to multiple reviewers that are using 3200Mhz kits.

This is another debate to open up. Just what speed ram should be used for testing? For example, Anandtech policy is to use the officially supported ram speed of the CPU/platform. Anything above that is overclocking, and where do you draw the line on what to overclock or how much? I think the main reason many don't consider high speed ram to be an overclock is the relative ease of buying a high rated kit, turn on XMP and more often than not it probably works. It doesn't always work, and it still catches people out.

 

Even if we limit the scope to PC gaming market, so we exclude office PCs for example, how much of that market actually runs above spec ram speeds? Reviewers have limited time so testing every possible variation is not going to happen from the start. IMO any review should include a reference point at official speeds, and if they choose to, include something higher to indicate possible improvement. I don't know if anyone has done a wider ram speed scaling test for Zen 3 yet, but I'll look forward to seeing one. 

 

1 hour ago, CarlBar said:

Once the GPU bottlenecks it comes down to how 1% and 0.1% low performance impacts average framerate. Intel does better in that department though it appears to be heavily RAM influenced.

Intel marketing: Best gaming 1% low results of any CPU, on every other Thursday, for game titles that have an even number of letters in its name.

 

I don't think they're desperate enough to go that route, but I couldn't resist  

[TetraSky]

It's worth nothing that the RAM Linus used, is indeed faster than the RAM GN used in their test.

3600 cl14 vs 3200 cl14. This is a latency of 7.77ns vs 8.5ns. 

(Not to mention, GN used 4x8GB which may introduce additional latency/burden on the memory controller, compared to 2x8GB)

 

There might be other factor at play, but it could explain why the LTT F1 results are so much closer/better than others.

Though it's kinda hard comparing both benchmarks when LTT tested at the highest preset, but GN used Medium for lord knows why. Not to mention they all used 1080p.

I get that they want to test the "cpu" and not the "gpu", but if they just crank it up to the highest possible setting, once they start getting CPU bottlenecked, we can still see the difference in performances between the various CPUs if they use the same GPU, no?

[RacA]

16 minutes ago, TetraSky said:

I get that they want to test the "cpu" and not the "gpu", but if they just crank it up to the highest possible setting, once they start getting CPU bottlenecked, we can still see the difference in performances between the various CPUs if they use the same GPU, no?

The higher the settings the more GPU bottleneck can be introduced to the benchmark, based on a specific game.
When GPU starts to be limiting factor we're seeing flawed results, in this case, we're not letting Zen 3 achieve it's max performance, and it's average FPS value ends up being lower.

[Taf the Ghost]

There's an art to teasing out CPU issues with Games. You have to land your settings in the right place where you're not just basically doing a GPU Driver specific Latency test (that's why 720p benchmarks end up being with highend GPUs). Because too low of res/settings and you're not really testing anything of note, but too much and you run into hard GPU bottlenecks. Each game is a little different, but somewhere in the 1080p/Medium to 1440p/high for each game will generally do it. Gaming Performance is still very much about draw calls & latency more than any actual IPC.

 

Most of this got kicked off by the TechPowerUp's results, which look a lot like some where GPU driver issue they need to test. Also, both AMD & Intel are very memory sensitive in high-end gaming. So memory settings used is something to be mindful of.

[bcredeur97]

if i had to guess this may come down to an optimization issue that will probably be resolved in the near future? For a brand new cpu, especially ones with major arch changes, I expect to have some oddness.

[porina]

1 hour ago, RacA said:

When GPU starts to be limiting factor we're seeing flawed results, in this case, we're not letting Zen 3 achieve it's max performance, and it's average FPS value ends up being lower.

The results are not flawed as such. They represent what you get in a mixed CPU/GPU (and other components) environment. The flaw is trying to read such results as a CPU-only indication of performance, where to get close to that you have to run unrealistic scenarios at which point the value needs to be questioned. If you want a pure CPU test, then pick a pure CPU test. That's why some love Cinebench, because it scales very well with CPU cores/threads/IPC and is hardly impacted by ram.

[ravenshrike]

4 hours ago, leadeater said:

I'm sure if there is enough noise GN Steve will do a bunch of tests across ram speeds and timing, and IF clocks, to see how that impacts things.

Didn't he mention offhand in either the 5950 or 5900 review that they would later be doing memory testing?

[Doobeedoo]

Would be interesting to see tests with various RAM kits tested side by side both sub 1080p and 1080p low and high too. Just to have more detailed picture.

[Nathanpete]

I honestly feel like the standardized kit of RAM any tech reviewer should consider using should be a 3000 or 3200 MHz kit of either 2x16GB or 4x8GB at a low CL. Anymore than that and you are giving AMD an unfair advantage. But if you start using 2666 MHz RAM then that is not an accurate representation of what PC and gaming enthusiasts will be buying. I believe both CPUs are on a more even playing field when the test bench has 3000 MHz RAM and I don't believe it will start skewing results. Besides, we are gamers looking at gaming benchmarks, so we will avoid 2666 MHz like the plague. 

 

Sure your performance improves on both sides if you increase RAM speeds, but we all know that the IF clock can play an important role in Ryzen performance. 

I am normally team Red, but I will give Intel the benefit of the doubt. Besides I start to ignore reviews that don't use a fair test bench or seem to have flawed testing methodology. 

[WereCat]

update

 

[Brooksie359]

2 hours ago, Nathanpete said:

I honestly feel like the standardized kit of RAM any tech reviewer should consider using should be a 3000 or 3200 MHz kit of either 2x16GB or 4x8GB at a low CL. Anymore than that and you are giving AMD an unfair advantage. But if you start using 2666 MHz RAM then that is not an accurate representation of what PC and gaming enthusiasts will be buying. I believe both CPUs are on a more even playing field when the test bench has 3000 MHz RAM and I don't believe it will start skewing results. Besides, we are gamers looking at gaming benchmarks, so we will avoid 2666 MHz like the plague. 

 

Sure your performance improves on both sides if you increase RAM speeds, but we all know that the IF clock can play an important role in Ryzen performance. 

I am normally team Red, but I will give Intel the benefit of the doubt. Besides I start to ignore reviews that don't use a fair test bench or seem to have flawed testing methodology. 

Most people buy 3200 at the least because anything lower doesn't even make sense as you can get a 3200 kit for super cheap. If people test with anything under 3200 than I would have to question the testing methodology as it would be purposely hampering performance. You can even find 3600 mhz kits for super affordable prices so idk why anyone would be upset about testing with those kits. If it was an excessive speed like 4000 mhz which cost significantly more and most won't buy it then I could understand but with 3600 you can easily buy a kit for around the same price as lower clocked kits. 

[williamcll]

Yeah I remember that some reviews benchmarked CSGO to hit 1200FPSMax 720AVG frames a second on 1080p.

Seems like memory latency is more important over memory clocks for this generation.

[Nathanpete]

8 hours ago, Brooksie359 said:

Most people buy 3200 at the least because anything lower doesn't even make sense as you can get a 3200 kit for super cheap. If people test with anything under 3200 than I would have to question the testing methodology as it would be purposely hampering performance. You can even find 3600 mhz kits for super affordable prices so idk why anyone would be upset about testing with those kits. If it was an excessive speed like 4000 mhz which cost significantly more and most won't buy it then I could understand but with 3600 you can easily buy a kit for around the same price as lower clocked kits. 

Did you not consider the fact that 3600 MHz give AMD an unfair advantage over Intel. I'm not team blue, I just want an even playing field. 3200 MHz is a more realistic option for gamers but I believe it would still give AMD a small unfair advantage in the infinity fabric. Hence why I also suggested 3000 MHz. And a small unfair advantage is significant when we make our buying decisions based on performance differences of like 2%. 

[RejZoR]

1 minute ago, Nathanpete said:

Did you not consider the fact that 3600 MHz give AMD an unfair advantage over Intel. I'm not team blue, I just want an even playing field. 3200 MHz is a more realistic option for gamers but I believe it would still give AMD a small unfair advantage in the infinity fabric. Hence why I also suggested 3000 MHz. And a small unfair advantage is significant when we make our buying decisions based on performance differences of like 2%. 

You're making it sound like Intel doesn't benefit from RAM at all. Which brings up the thing, has AMD been nerfed the entire time because of poor choice of RAM by testers? Shit swings both ways...

[porina]

27 minutes ago, RejZoR said:

You're making it sound like Intel doesn't benefit from RAM at all. Which brings up the thing, has AMD been nerfed the entire time because of poor choice of RAM by testers? Shit swings both ways...

AMD still only officially support 3200. Anything above that is an overclock. If it is relatively easy and reachable to run faster ram, it is fine to run faster on both sides, but baseline (supported) speeds should also be a reference point. If anything it is AMD that is pushing it a bit, since they openly stated 3600 as the performance sweet spot for Zen 2, so that was a speed grade people gravitated towards.

[RejZoR]

2 hours ago, porina said:

AMD still only officially support 3200. Anything above that is an overclock. If it is relatively easy and reachable to run faster ram, it is fine to run faster on both sides, but baseline (supported) speeds should also be a reference point. If anything it is AMD that is pushing it a bit, since they openly stated 3600 as the performance sweet spot for Zen 2, so that was a speed grade people gravitated towards.

I mean, X99 also officially supported 2400MHz. How many people actually ran that? If price difference is not big, it's natural to take faster one with lower timings.

[terminalinfinity]

Im pretty sure I found the reason for the wild review swings

There is a MAJOR issue I reported and confirmed about the MSI BIOS here

Basically the jist is the current iteration of MSI's 500 series BIOS causes WHEA errors W/ Zen 3 above 3200 MHz memory frequency and memory latency above 100ns (Should be sub-70 at least).

Looking at the reviews with major deviation like hardware unboxed and anandtech, they all share one thing in common: MSI motherboards

 

Issue found.  The deviance is caused by a buggy BIOS.

These benchmarks need to be re-done on non-MSI motherboards

[Brooksie359]

8 hours ago, Nathanpete said:

Did you not consider the fact that 3600 MHz give AMD an unfair advantage over Intel. I'm not team blue, I just want an even playing field. 3200 MHz is a more realistic option for gamers but I believe it would still give AMD a small unfair advantage in the infinity fabric. Hence why I also suggested 3000 MHz. And a small unfair advantage is significant when we make our buying decisions based on performance differences of like 2%. 

No it wouldn't. That's not giving an unfair advantage when you can buy 3600 mhz memory for around the same price as lower speed kits. If you are running lower speed kits just to make sure amd doesn't perform as well that would be giving intel an unfair advantage. Idk why you feel that reviewers should purposely hamper performance of amd chips that way people are misinformed and more likely to buy intel rather than know the true potential of amd chips. You have to realize most of the people watching these videos are tech enthusiasts and are more than likely going to buy 3600 mhz kits because it is known to be good for amd. 

[leadeater]

2 hours ago, Brooksie359 said:

No it wouldn't. That's not giving an unfair advantage when you can buy 3600 mhz memory for around the same price as lower speed kits. If you are running lower speed kits just to make sure amd doesn't perform as well that would be giving intel an unfair advantage. Idk why you feel that reviewers should purposely hamper performance of amd chips that way people are misinformed and more likely to buy intel rather than know the true potential of amd chips. You have to realize most of the people watching these videos are tech enthusiasts and are more than likely going to buy 3600 mhz kits because it is known to be good for amd. 

Additionally reviewers in the past have gone through the effort to find the most optimal memory configurations for Intel systems and have standardized on what is known to be overall the best ram configuration for an Intel system. It is totally fair thing to do for AMD so they also know what to standardize on for those too. It's not like reviewers just choose at random or guess which ram to use, no they actually tested it or looked at someone else who has and used that information to create their standardized test system.

 

If using faster ram in an Intel system resulted in performance above margin of error they would be using it, reviewers are not short on ram or vendors willing to supply it to them. 

[LAwLz]

Linustechtips.com, where not running AMD at overclocked settings and Intel at stock is "unfair"... 

 

Maybe people should dial back their fanboyism a little? You're literally advocating for using overclocked settings on one platform but not the other. 

Why not agree upon using the highest clocked RAM supported by both platforms? At least for the" stock" tests.