UserBenchmark Adjusts Algorthym to Favor Single-Thread Performance, Causing Strange Results

[infinitytec]

UserBenchmark.com has adjusted the algorithm for how the overall performance of a CPU is calculated.

 

Previously, single-core performance made up 30% of the result, quad-core made up 60%, and the final 10% was reserved for "multicore" performance. This was calculated so that an Intel Core i7-7700K would equal approximately 100% score.

 

The new scoring places the i9-9900K at the 100% mark, but also changes the makeup of the balancing of the scores. Single-core scoring now accounts for 40% of the result, and quad-core score takes a hit down to providing 57% of the score. Finally, the multicore score (specified to be 64 threads) only makes up 2% of the final score.

 

UserBenchmark claims that this is to better reflect the importance of single-threaded performance in gaming. However, more workloads have been embracing more threads. The balance change makes the results for effective CPU speed seem off, namely when it comes to AMD CPUs and even Intel CPUs that have more threads, but may lag behind on single-threaded performance.

 

For example, comparing the Ryzen 9 3900X to the Core i3-9350KF yields a mere margin of 3% in favor of the Ryzen 9. This is comparing a twelve-core 24-thread chip to a four-core chip.

 

However, comparing the "Real World Speed" tells a different story. The lowest improvement the tests show the Ryzen 9 having is 2%, but the largest gain is a whopping 85%.

 

And now some before-and-after with the Ryzen 5 3600:

 

(before)

 

(after)

 

The Average Bench drops 15.6%. Granted, some of this drop could be because of the change of reference CPU. But the effective speed, average user bench, and peak overclock bench all take major hits.

 

Intel has a similar decrease, though not to as much of a degree, on the i7-8700K:

(before)

 

(after)

 

 

And about the i9-9900K being at 100%, I believe it fair to say that it has more than 9% effective speed than an i3-9350KF.

 

The change in results has caused some uproar, especially among those who are fans of AMD, as this change seems to punish CPUs with higher core counts, which AMD is excelling at.

 

 

 

 

UPDATE 7/26/2019: UserBenchmark has addressed concerns... by brushing them off at best.

The UserBenchmark FAQ page on the effective CPU speed has been updated. (Archived version here)

 

They specifically address the AMD community's displeasure with the change:

Quote

AMD community

Shortly after the Ryzen 3000 release, which we welcomed emphatically, we noticed that our CPU gaming and desktop estimates were unrealistically overestimating CPUs with core counts beyond 8 so we corrected them. The underlying data points for single, quad and multi core performance remain unchanged and are clearly visible together with gaming, desktop and workstation scores on each of our pages. Back in the early days of the AMD FX-8350 our effective speed index was predominantly single core and at that time we were heavily lobbied with cries of "cores are only getting more and more relevant". We are constantly tuning and updating our effective speed indices to match the latest developments and will continue to do so independently.

So they noticed an issue and "fixed" it?

This seems to say otherwise.

 

But it doesn't stop there:

Quote

Presently we are aware that we slightly overestimate the latest batch of AMD 5700 graphics cards but, unsurprisingly, nobody is crying fire and continuously spamming us about that.

Good job, UserBenchmark. You admitted what we have known for a long time: you are only good for estimations. But, more importantly, you are admitting your tests have flaws. Why should we not be upset when your recommendations are saying we can all go get i3s and compare them to Ryzen 9s? People are not going to be concerned with a slight test inconsistency. People are going to be concerned when tests are poorly balanced across all products.

 

 

And one last bit on their "improved" FAQ page:

Quote

Finally ...

Beware the organized army of shills who pump one brand or another and deal in hot air rather than reason. Only use trusted independent sources to know your chops before parting with any of your hard earned cash. May the force be with you.

Granted, something similar was on the last version. I think that the concerns the community has are reasonable. People who don't know better may waste money based on this site. And, to top it off, a nice video of how some people put too much fat on sheep before slaughtering them.

 

[emosun]

I like how slowly over the last decade cpus are adding more and more cores to solve the speed problem..... yet somehow theres this small very vocal few who think a quad core is the cpu of the future.

I think userbench is catering to the mostly gamer user base as opposed to really measuring hardware speed.

[Jurrunio]

I thought no one checks the "effective speed" coloum and go straight down below to the individual tests? 

[Arika]

Userbench should not be used for any serious comparison, it's good to get a general idea, but you should ALWAYS go and find full reviews on both products and compare them yourself

[wanderingfool2]

21 minutes ago, emosun said:

I like how slowly over the last decade cpus are adding more and more cores to solve the speed problem..... yet somehow theres this small very vocal few who think a quad core is the cpu of the future.

I think userbench is catering to the mostly gamer user base as opposed to really measuring hardware speed.

It all depends on what you consider to be "hardware speed", and what your usecase is.  Most day to day tasks (which accounts for most general consumers and ones that may be looking for comparisons I would argue don't require much more than 4 cores)...actually "hardware speed" is probably the wrong term to use that case...as a single core performance in my opinion is the measurement of hardware speed (vs parallelism).

[emosun]

1 minute ago, wanderingfool2 said:

It all depends on what you consider to be "hardware speed", and what your usecase is.  Most day to day tasks (which accounts for most general consumers and ones that may be looking for comparisons I would argue don't require much more than 4 cores)...actually "hardware speed" is probably the wrong term to use that case...as a single core performance in my opinion is the measurement of hardware speed (vs parallelism).

ah see I don't do anything that requires a single cpu core short of using notepad and ms paint so that's where we'll definitely differ

[poochyena]

You mean a 4 core isn't better than a 32 core CPU?
https://cpu.userbenchmark.com/Compare/Intel-Core-i3-9100F-vs-AMD-Ryzen-TR-2990WX/m796379vsm560423

 

[cj09beira]

in a time where things are mostly multithreaded they think its a good idea to go the other way

 

 

@emosun revy  

 

[wanderingfool2]

1 hour ago, emosun said:

ah see I don't do anything that requires a single cpu core short of using notepad and ms paint so that's where we'll definitely differ

It all depends on workload really.  The way I look at it, I can see the merits of putting more weight on single core performance, because in my opinion the larger majority (and not talking about tech people here), will likely not require as many threads (and single core performance is likely to make things feel snappier).

 

An example being people who may utilize excel...it typically is stuck to one core....or when using a web-browser, while it may utilize more threads it is very unlikely to utilize all of the threads (and in some real world cases I found it more pinning a single core with the others doing smaller tasks).  So I am just saying I can see merit in the fact of weighting single core performance heavily.  (That isn't to say I would prefer a faster 2 core over 4 core vs 8 core...but I think with most people here being more tech oriented there is a tendency to forget that the non-techy people I think outnumber us)

[Nowak]

A joke.

[melete]

15 minutes ago, wanderingfool2 said:

It all depends on workload really.  The way I look at it, I can see the merits of putting more weight on single core performance, because in my opinion the larger majority (and not talking about tech people here), will likely not require as many threads (and single core performance is likely to make things feel snappier).

 

An example being people who may utilize excel...it typically is stuck to one core....or when using a web-browser, while it may utilize more threads it is very unlikely to utilize all of the threads (and in some real world cases I found it more pinning a single core with the others doing smaller tasks).  So I am just saying I can see merit in the fact of weighting single core performance heavily.  (That isn't to say I would prefer a faster 2 core over 4 core vs 8 core...but I think with most people here being more tech oriented there is a tendency to forget that the non-techy people I think outnumber us)

I do wonder how many people choose CPUs for Excel or web browsing by looking at UserBenchmark, though. My guess is that their audience is heavily slanted towards enthusiasts who are either doing other work that's more CPU intensive (be that gaming, rendering, video editing, compiling, etc.) or just chasing performance for the sake of performance.

[wanderingfool2]

2 minutes ago, melete said:

I do wonder how many people choose CPUs for Excel or web browsing by looking at UserBenchmark, though. My guess is that their audience is heavily slanted towards enthusiasts who are either doing other work that's more CPU intensive (be that gaming, rendering, video editing, compiling, etc.) or just chasing performance for the sake of performance.

Those are two examples, to show that a single core could have benefits to everyday users.  While you may guess that the audience is heavily towards enthusiasts, any enthusiast would know to look at more than just the end score.  Only day to day consumes (ie those who don't know what a 2 core vs 4 core is) I think would use the overall score number (and I do think that putting a heavier focus on single core is better).

 

Again, lets be honest, enthusiasts or even semi-techy people would scroll down a bit and find the numbers that are more practical for them

[melete]

6 minutes ago, wanderingfool2 said:

Those are two examples, to show that a single core could have benefits to everyday users.  While you may guess that the audience is heavily towards enthusiasts, any enthusiast would know to look at more than just the end score.  Only day to day consumes (ie those who don't know what a 2 core vs 4 core is) I think would use the overall score number (and I do think that putting a heavier focus on single core is better).

 

Again, lets be honest, enthusiasts or even semi-techy people would scroll down a bit and find the numbers that are more practical for them

Yeah, I do think knowledgeable enthusiasts would do that but I think UserBenchmark is also kind of the lowest common denominator for PC enthusiasts. People who are young or inexperienced in the hardware world, or just want a gaming or workstation PC, or are just starting to learn about PC hardware will go there to look at CPUs. I bet most of us here would rather read a CPU review to take a look at their benchmarks instead of going off anything UserBenchmark has to offer, regardless of how they calculate their scores. It's the people who don't quite have that sophistication or knowledge that get hurt by strange CPU score weighting, I think.

[poochyena]

2 hours ago, Arika S said:

you should ALWAYS go and find full reviews on both products and compare them yourself

How do I compare my 6600k to the new ryzen 3700 myself? Or compare my old laptop's cpu to a desktop cpu?

[SenioRR]

I mean the site was already a very rough way of estimating performance, although quiet convenient so a lot of new users would look at UserBenchmark.

I mean just look at this:

https://cpu.userbenchmark.com/Compare/Intel-Core-i3-8350K-vs-AMD-Ryzen-TR-2990WX/3935vsm560423

Or this:

https://cpu.userbenchmark.com/Compare/Intel-Core-i7-4770K-vs-Intel-Core-i5-4670K/1537vs153

Even this!

https://cpu.userbenchmark.com/Compare/Intel-Core-i3-9350KF-vs-Intel-Core-i9-9980XE/m775825vsm652504

 

Either the people behind UserBenchmark have lost all of their brain cells OR there was a sizeable cheque sent their way.

Anyway, I usually go to GamersNexus for benchmarks but an upcoming PC enthusiast might look at UserBenchmark as a reliable source of information on part performance.

[Arika]

13 minutes ago, poochyena said:

How do I compare my 6600k to the new ryzen 3700 myself?

2 hours ago, Arika S said:

go and find full reviews on both products

 

 

14 minutes ago, poochyena said:

Or compare my old laptop's cpu to a desktop cpu?

Why do you need anything more than a general idea when comparing old stuff? I'm not saying don't use Userbench, i'm saying don't use it as gospel.

[5x5]

Smells like 2004-2005 all over again

[porina]

The eternal problem is that people try to simplify down complex measures into a single number. That single number can never usefully represent everyone's use case. They have to pick one scenario, and you either agree with it or disagree.

 

This is a site I've never paid much attention to anyway, since even when they do take measures it isn't really presented in any usable format. 

[Neftex]

smells like intel marketing had a talk with userbenchmark, maybe even forgot a wallet on the table when leaving

[jagdtigger]

15 minutes ago, Neftex said:

smells like intel marketing had a talk with userbenchmark, maybe even forgot a wallet on the table when leaving

My thoughts exactly. The only remaining advantage intel has is single core. This is way too much to be a coincidence.....

[Dabombinable]

Even fucking Skyrim can use 4 cores...so 8 years later, how is single threaded performance that important?

[RejZoR]

1 hour ago, jagdtigger said:

My thoughts exactly. The only remaining advantage intel has is single core. This is way too much to be a coincidence.....

That's not even true anymore with Ryzen 3000. Sure it varies depening on compiler used or workload, but 3900X has the same single thread performance as 9900K even at 400-500MHz clock difference.

[SenioRR]

Considering Intel's history of unethical agreements with the likes of Dell, HP and others just to put AMD at a disadvantage, I think it is clear that they have been paid to change the formula so that it favors Intel.

[Blademaster91]

Except if Intel were involved in Userbench being so skewed towards fewer cores, they wouldn't be showing the as already mentioned result of a i3-9350KF being "better" than a i9-9980XE.

I've never found Userbench very useful though, even before they changed the scoring, it was obviously a site someone shouldn't use to make a decision without looking at real benchmarks first.

[Sauron]

It's almost as though benchmark aggregators are a stupid way of comparing cpus and shouldn't be trusted.