Passmark CPU Scores Significantly Changed?

[harbinrepairs]

Has anyone noticed that a lot of the CPU benchmark scores on Passmark have changed significantly. For the past few years the i7-4790k (almost 6 year old CPU) has always had a score around 11,000 and for some reason this week it has dropped to around 8600 as of March 5th, 2020. That's a 22% difference compared to the scores that have stayed the same for almost 6 years, I took a screenshot from Wayback Machine in December 2019 to show the difference. This is kind of frustrating because I use this a lot when comparing CPU's in my customer's PC's with newer ones to rule out whether or not it's worth upgrading older hardware. Any thoughts on this? Does anyone know if Passmark has changed their benchmarking algorithm lately that might be affecting this? I also did notice there seems to be a much higher score difference (22%) with 4th gen and older Intel CPU's, and only a 4%-15% difference with newer gen intel and AMD CPU scores.

 

Current score (March 5th, 2020): https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i7-4790K+%40+4.00GHz&id=2275

 

Wayback Machine capture from December 2019: https://web.archive.org/web/20191212215318/https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i7-4790K+%40+4.00GHz&id=2275

[rogerswb]

Yeah, I noticed it too. I've been doing a lot of used CPU shopping and found a number of them that I was interested before they started changing, but when I went back to check them out yesterday, quite a few of them were scoring significantly lower. I haven't found an explanation for it anywhere. It's not all happening at one time, either. My i7-3820 was rated at 8990 back in October, and I swear that I thought it was around 7400 yesterday (though I don't have anything to prove that). It's 6000 today.

[SirTanon]

I was just noticing that myself, and I find it very disconcerting, as well as frustrating.  Like rogerswb, I have been looking at a whole bunch of used CPUs to start doing some builds - some Xeons, lots of older core i3/5/7 cpus, etc.. even some AMDs, and I use my current AMD FX-8350 as the point of reference from which to compare.   Wayback machine shows that just a week ago - 03/09 - it had a score of 8949, but now, today, it has a score of 6246.  That's a 30.2% drop.

 

Similarly, the Xeon E5620, a CPU I've actually recently bought quite a few of, was scoring 4770 12/15/19, and today scores just 3103.  That's a 35% drop.

 

The Xeon X5570, back on December 11, 2019 scored 5615, but now scores just 3640.  That's a 35.2% drop.

 

E5-2620 - On 12/12/19 scored 7904, and today 5138 - a 35% drop.

 

i3-550 - On 12/21/19 scored 2829, and today 1687 - a 40.4% drop.

 

Core 2 Quad Q6600 - December 11, 2019 got a 2946, and today scores just 1590.  That's a whopping 46% drop.

 

Core i9-9980XE got a 29,576 on 12/15/19, but a 28,160 today.  4.8% drop.

 

Strangely, the AMD Athlon 200GE does not tell the same story.  On 12/13/19, it scored a 4952, and today scores a 4482.  Only a 9.5% drop.

 

.. and the Threadripper 3970X got 61,454 on March 12, 2020, and scored almost the same - 60,980 today.  That's less than a 1% drop.  I might give this one a pass, however, as the sample counts are low.  On 3/5/2020, it only had a score of 48,526 with a sample size of 72, vs a sample size of 84 on 3/12.

 

And then there's a really odd one - The Ryzen 7 2700X got a score of 16,949 on December 13, 2019, but went UP to 17,970 as of today.  That's an INCREASE of 6%!

 

What on earth could be behind these wild, and seemingly inconsistent changes?   It almost seems like someone has been running a TON of new benchmarks with a ton of these CPUs using crippled configurations.   Case in point - FX-8350 on 3/9/20 vs on 12/15/19.  Note the sample counts - a change of nearly 1500, in just 1 week.

 

[thri1hous]

i have noticed this as well. has anyone figured out what is going on?  maybe some update fix for some vulnerability resulted in a performance hit?

[papajo]

Never used passmark because I think there are more reliable metrics.. but what I think is going on is that with the advent of new CPUs which are better both in multi threading and in single thread applications the averages gone higher...

[thri1hous]

does passmark score a cpu against the average score or something? I would have expected it to maybe take the average of all the like SKUs but then as new CPUs come out they would just have increasingly higher scores.

maybe there are more reliable metrics but people still use passmark. for example plex recommends following a "rough guideline" for cpu selection https://support.plex.tv/articles/201774043-what-kind-of-cpu-do-i-need-for-my-server/

basically you need a cpu with a passmark score of about 2000 to transcode one 1080p stream. so if new CPUs retroactively change an old CPUs score by influencing the average then devs would have to be constantly reevaluating their CPU recommendations.

[whitewolf60]

I can only say "Me too"; you're definitely not imagining things... my God, look at the differences in the single-thread scores in the original post, and the small change in total samples between the two instances.

 

I've been dealing mostly with Xeons, and had the scores of the ones I most commonly used committed to memory; not only that, but I have kept many scores on a spreadsheet. Yes, I've suddenly noticed drops of around 30% on most everything. My first assumption is that they must have changed the benchmarking standards, and retroactively factored scores from before the change. More/faster new processors shouldn't change benchmarks on OLDER processors; as I understand it, only new samples of IDENTICAL processors would have the ability to do that, and you wouldn't expect a small number of new samples to do that so radically unless there was some hanky-panky going on.

 

Examples: E5-2687W that WAS 14341 is NOW 7526; E5-2690 that WAS 13723 is NOW 9970. WTF? Those are MASSIVE changes (the 2687W has dropped nearly FIFTY percent!), and note also that, whereas the E5-2687W seemed to have a clear advantage previously, now the E5-2690 seems to have an even GREATER advantage over the 2687W. I feel like I'm now comparing apples to oranges.

 

I fully understand why individual samples can be all over the map, but how do AVERAGES that had previously been stable SUDDENLY and almost ACROSS THE BOARD drop so precipitously? Note that I have seen it even in low-power 2nd, 3rd, and 4th gen. i5s, for example (4200U WAS 3292, NOW 2193).

 

Maybe one of us needs to contact Passmark...

[Guest]

8 minutes ago, whitewolf60 said:

\

 

Maybe one of us needs to contact Passmark...

Or just don't use it.....

[whitewolf60]

11 minutes ago, ShrimpBrime said:

Or just don't use it.....

It has previously been sufficient for my purposes (note that I don't use the software, only the charts), but this anomaly needs explanation. Of course, I don't do any "gaming" systems or anything like that. As I've told people before: "I calculated how much money I've wasted on computer games, and it was 25-cents, because I played Pong in an arcade once back in the early '70s" : )

 

I'm more a fan of reality...

[Guest]

9 hours ago, whitewolf60 said:

It has previously been sufficient for my purposes (note that I don't use the software, only the charts), but this anomaly needs explanation. Of course, I don't do any "gaming" systems or anything like that. As I've told people before: "I calculated how much money I've wasted on computer games, and it was 25-cents, because I played Pong in an arcade once back in the early '70s" : )

 

I'm more a fan of reality...

Yes well truth be told, it was never the most consistant way to bench a system in the first place. 

Results have always varied.

 

So they make a change in algorithm to accomidate processors and software of today. Now this is a guess really, but would explain well why the sudden change in scoring and points.

 

HWBot has done this almost every revision, but at least most of the benchmarks are consistant...... Just the point system was changed.

[thri1hous]

ok so i emailled passmark and they got back to me fast. sorry im just now updating this.

 

basically I sent this out

Quote

Hello,

 

I my self as well as a few other people in the enthusiast PC community have noticed that a lot of CPU benchmarks have changed recently for older CPUs. her is an example below 

this issue has been documented in much more detail here on this forum https://linustechtips.com/main/topic/1162222-passmark-cpu-scores-significantly-changed/

 

We would much appreciate an explanation to what is going on here. has the algorithm changed? has there been a lot of new samples added to the database? have new CPU releases some how retroactively changed the average or something?

 

We would all appreciate you guys shedding some light on the issue 

then they responded with this

 

Quote

Hi,

Thanks for the Email.

Yes, big algorithm changes.

You can find a detailed summary of the changes here,
https://www.passmark.com/support/performancetest_faq.php#V10Results

Obviously we want to start using the PT10 results on our graphs. But if we wait until we have a million PT10 results, that might take a year. And in the meantime we have no results for any new CPUs on PT9, as noboby will be using PT9 anymore.

So the solution we selected (the least worst solution from the collection bad solutions) was to take all the average CPUMark values from PT9 (one value per CPU model) and then start averaging that with all the new PT10 results as they come in. So what this means, especially for the first few weeks is a lot of volatility as the graphs slowly move to reflect more of the PT10 result and less of the PT9 results. Initial PT10 results have a big impact, but each additional PT10 result has less impact as a new average is found (which has already happened for all but the very rare CPUs).

Kind regards,
David Wren
PassMark Software
Kind Regards
Level 5, 63 Foveaux St.
Surry Hills, 2010
Sydney, Australia
P: + 61 2 9690 0444
F: + 61 2 9690 0445
W: www.passmark.com

 

so yeah basically algorithms changed. now I need to ask plex to reassess their spec requirements.