A new CPU benchmark

[FJLJ]

Hello all!

 

Long time LTT fan here....

 

One thing I have began to notice in the industry of bench-marking... is that when it comes to testing a single specific piece of hardware... it doesn't really exist.

 

You might start scratching you head here thinking... yes there does... but let me explain.

 

Many benchmarks are created to task a machine in a specific way that yields some type of measurable results. However at what cost in overhead? What other hardware is being utilized? What factors are really at play here?

 

I spent hours wracking my brains on how, at the machine level I could measurably stress JUST the CPU, in a manner that would provide useful results that were relative to real world application, and not just some synthetic test score.

 

and... this is what i came up with...

			0x31,0xC0,0x31,0xC9,0x31,0xD2,0x31,0xDB,0xE8,0x00,0x00,0x00,0x00,0x5E,0xE8,0x00,
			0x00,0x00,0x00,0x40,0x48,0x40,0x48,0x40,0x75,0xF9,0x83,0xC0,0x02,0x83,0xE8,0x01,
			0x83,0xC0,0x03,0x83,0xE8,0x05,0x83,0xC0,0x02,0x75,0xEF,0x41,0x8D,0x04,0x8D,0xFE,
			0xFF,0xFF,0xFF,0x8D,0x44,0x49,0xFF,0x8D,0x04,0x09,0x75,0xEF,0x91,0x87,0xCA,0x87,
			0xD3,0x93,0x40,0x75,0xF7,0x83,0xC0,0x01,0x83,0xC1,0x03,0x83,0xC3,0x02,0x0F,0x6E,
			0xC0,0x0F,0x6E,0xC9,0x0F,0x6E,0xD3,0xF3,0x0F,0xD6,0xC0,0xF3,0x0F,0xD6,0xC9,0xF3,
			0x0F,0xD6,0xD2,0x66,0x0F,0xFE,0xC1,0x66,0x0F,0xFA,0xC2,0x66,0x0F,0xFE,0xC1,0x66,
			0x0F,0xFA,0xC1,0xF2,0x0F,0xD6,0xC0,0x0F,0x7E,0xC0,0x21,0xC0,0x75,0xE5,0x0F,0x77,
			0xD9,0xE8,0xDE,0x8E,0xDD,0x00,0x00,0x00,0xDE,0xB6,0xDF,0x00,0x00,0x00,0xDE,0x8E,
			0xDD,0x00,0x00,0x00,0xDE,0xB6,0xDF,0x00,0x00,0x00,0xDF,0xE0,0x24,0x08,0x74,0xE2,
			0xDB,0xE3,0xD9,0xE8,0xD8,0x8E,0xE5,0x00,0x00,0x00,0xD8,0xB6,0xE1,0x00,0x00,0x00,
			0xD8,0x8E,0xE5,0x00,0x00,0x00,0xD8,0xB6,0xE1,0x00,0x00,0x00,0xDF,0xE0,0x24,0x08,
			0x74,0xE2,0xDB,0xE3,0x31,0xC0,0x40,0x0F,0xA2,0x31,0xC0,0xC1,0xC9,0x1A,0x73,0x13,
			0x66,0x0F,0x3A,0xDF,0xC8,0x03,0x66,0x0F,0x38,0xDC,0xC1,0x66,0x0F,0x38,0xDC,0xC8,
			0x40,0x75,0xF3,0x31,0xC9,0x31,0xD2,0x31,0xDB,0xC3,0x03,0x00,0x02,0x00,0xA7,0x01,
			0x80,0x3F,0x4F,0x02,0x80,0x3F

This may not look like code to most... yet old-school programmers and those who have an understanding of machine code or even x86 assembly will understand what this does.

 

However this day in age implementation is everything... and console applications are not typically met with popularity... SOOO...

I had to actually make a GUI..

  

 

Now I know it is not exactly flashy or have a ton of bells and whistles... Heck even the "progress bar" is a static animation... ("i know lame but there is a reason...).

 

AND JUST WHAT IS THAT REASON FRANK?

Well the GUI is there to give the user an easy way to run the benchmark and see/submit/share their results.. NOT WASTE CPU DISPLAYING A PROGRESS BAR!!!

 

SOOO HOW DOES THIS WORK FRANK?

The above assembly is ran.

oh you want more... fine..

 

The C# GUI program determines the number of cores and how many instances and on which cores the code will run to determine a baseline expected CPU utilization %.

The CPU % is monitored and logged, the assembly code has 1 section in it that will cause a specific increase in CPU % at a specific instruction that can be calculated.

When the code completes the results are spit out, the passively logged data is crunched.

The actual time points above the expected CPU % are compared to the expected, as well as the overall time above or below the expected CPU % is gathered.

This information is then used to calculate the overall performance in various cpu tasks that are found in various application types weighted by importance/frequency of use in applications.

The total score is the combined total of each of these tasks scores averages.

 

WHAT CAN BE EXPECTED FRANK?

well there is still a bit of work to do...

1. The website is awful... (just through something together quickly to get the submissions working...)

2. adding an information bubble to each of the difference scores...

3. what other recommendations do you have?

 

CAN WE TRY IT OUT FRANK?

if you really want to... you can find it at my very not good website https://give.academy/downloads/

 

CAN WE SEE CURRENT RESULTS?

sure.. again not too impressive the page currently just lists all submissions highest scores first (big changes comming) 

http://bit.ly/2zvoVoO (blog style crappy site... means long links... sorry..

(unshortened) https://give.academy/benchmark/2017/11/04/TruCpuBenchResults/

 

But really I would like all of your feedback, so together we can make the best and most reliable benchmarking software. The TruBench.

 

Thanks,

Frank

[Enderman]

So... sort of like linpack?

[FJLJ]

1 minute ago, Enderman said:

So... sort of like linpack?

Good question! Maybe I wasn't detailed enough.

 

TruCpuBench does not set out to solve anything or to calculate anything. it has a set of instructions that run in machine code at the CPU level. there are no decisions being made it just runs as fast as it can until is reaches the end (the secret to how it can do this without having to make any decisions is in the hexcode above for anyone who can/wants to read it. ) as it's a bit dense/dry to explain (maybe in a video. ).

 

Thanks,

Frank

[Tabs]

8 minutes ago, Enderman said:

So... sort of like linpack?

From looking at the code, there's absolutely zero way it will stress anything like SSE or advanced acceleration tech like AVX. Perhaps a "worst case scenario" benchmark? I can see the use, but many benchmarks nowadays are designed to emphasise the "edge case" performance improvements for software that takes advantage of these instructions, so I'm unsure how widespread a "vanilla" benchmark could become.

 

Edit: OP, I'm glad you said basically the same thing, but I'm also slightly concerned that my results are almost *identical* to the ones posted for a FX-8350 when I'm using a 6700. It doesn't seem to give realistic "expectations" of base performance.

Edited November 11, 2017 by Tabs
More info

[Enderman]

7 minutes ago, FJLJ said:

Good question! Maybe I wasn't detailed enough.

 

TruCpuBench does not set out to solve anything or to calculate anything. it has a set of instructions that run in machine code at the CPU level. there are no decisions being made it just runs as fast as it can until is reaches the end (the secret to how it can do this without having to make any decisions is in the hexcode above for anyone who can/wants to read it. ) as it's a bit dense/dry to explain (maybe in a video. ).

 

Thanks,

Frank

But part of knowing the CPU performance is how fast it can cache things, perform floating point operations, use branch prediction acceleration, and stuff like that.

Your test basically only tests a fraction of the CPU then.

[FJLJ]

6 minutes ago, Tabs said:

From looking at the code, there's absolutely zero way it will stress anything like SSE or advanced acceleration tech like AVX. Perhaps a "worst case scenario" benchmark? I can see the use, but many benchmarks nowadays are designed to emphasise the "edge case" performance improvements for software that takes advantage of these instructions, so I'm unsure how widespread a "vanilla" benchmark could become.

 

Edit: OP, I'm glad you said basically the same thing, but I'm also slightly concerned that my results are almost *identical* to the ones posted for a FX-8350 when I'm using a 6700. It doesn't seem to give realistic "expectations" of base performance.

Forgot to mention that in the original post.. Currently the results page is a bit mis-leading because it does not display the frequency the cpu was at. My 8350 was only barely able to get that result after closing EVERYTHING running and having an overclock to 4.75Ghz. 

 

Tabs, do you have any ideas on what I could add differentiate or make the results more "Readable" ? I was thinking of dividing the scores by the cpu frequency however this may yield some skewing in the results. 

 

[FJLJ]

2 minutes ago, Enderman said:

But part of knowing the CPU performance is how fast it can cache things, perform floating point operations, use branch prediction acceleration, and stuff like that.

Your test basically only tests a fraction of the CPU then.

Excellent concern, the test does test all the things you have mentioned even mis-use of them IE "web games using floating point variable FOR EVERYTHING even integer math". caching comes into play very quickly actually, by design it is created to assume all instructions will be in cache as they are happening, we could actually time the difference in segments to see how quickly the cache tosses out the last segment in favor of the next.  

 

It uses every register in the cpu, does fpu translations aes key rotation, mmx and xmm bit roatations as well. 

can you think of something I should add to the assembly code? I am here to learn what needs improved and what should be added.  thank you for being a part of that!

[Enderman]

1 minute ago, FJLJ said:

Excellent concern, the test does test all the things you have mentioned even mis-use of them IE "web games using floating point variable FOR EVERYTHING even integer math". caching comes into play very quickly actually, by design it is created to assume all instructions will be in cache as they are happening, we could actually time the difference in segments to see how quickly the cache tosses out the last segment in favor of the next.  

 

It uses every register in the cpu, does fpu translations aes key rotation, mmx and xmm bit roatations as well. 

can you think of something I should add to the assembly code? I am here to learn what needs improved and what should be added.  thank you for being a part of that!

No, I don't know enough about assembly to do anything  I transferred out of computer science.

[Tabs]

3 minutes ago, FJLJ said:

Forgot to mention that in the original post.. Currently the results page is a bit mis-leading because it does not display the frequency the cpu was at. My 8350 was only barely able to get that result after closing EVERYTHING running and having an overclock to 4.75Ghz. 

 

Tabs, do you have any ideas on what I could add differentiate or make the results more "Readable" ? I was thinking of dividing the scores by the cpu frequency however this may yield some skewing in the results. 

 

I think the problem when you start to think about stuff like that is that you'd have to account for system usage (ie, non-benchmark usage) as well as core utilisation.

 

When I ran the benchmark, it peaked at 90% utilisation (4 pegged cores - presumably physical - and 4 underutilised cores - presumably logical due to HT), and it wasn't "stressful" enough for my machine to utilise turbo speed bins. My cpu fans didn't even start ramping, though I didn't have temperature monitoring running at the time.

[FJLJ]

Just now, Enderman said:

No, I don't know enough about assembly to do anything  I transferred out of computer science.

no experience necessary, if you can think of an internal operation of a cpu that I may have missed or overlooked, please let me know and I will GLADLY add it into the bench. This is very early stages, essentially testable for 3 days, and would like to get a wide assortment of CPU results so I can work out what things may not be quite right or what things might need to be added.

[WereCat]

I will give it a try when I get home

 

Edit:

Does RAM speed/latency play any significant role in this benchmark?

[FJLJ]

2 minutes ago, Tabs said:

I think the problem when you start to think about stuff like that is that you'd have to account for system usage (ie, non-benchmark usage) as well as core utilisation.

 

When I ran the benchmark, it peaked at 90% utilisation (4 pegged cores - presumably physical - and 4 underutilised cores - presumably logical due to HT), and it wasn't "stressful" enough for my machine to utilise turbo speed bins. My cpu fans didn't even start ramping, though I didn't have temperature monitoring running at the time.

Also forgot to mention the benchmark does take into account the beginning usage and will adjust for discrepancies by a rolling amount of external differences. HOWEVER... boost/turbo...

THIS IS AN EXCELLENT POINT AND AN OVERSIGHT ON MY PART!!! THANK YOU!

 

I was thinking at the level of "if i peak out the CPU I will not be able to measure the expected peaks in usage." so I have it written to use 75%-88% depending on core count... as when I test on my machine at allowing 100% it locks EVERYTHING until its finished.

 

Thank you!

[Tabs]

14 minutes ago, FJLJ said:

Also forgot to mention the benchmark does take into account the beginning usage and will adjust for discrepancies by a rolling amount of external differences. HOWEVER... boost/turbo...

THIS IS AN EXCELLENT POINT AND AN OVERSIGHT ON MY PART!!! THANK YOU!

 

I was thinking at the level of "if i peak out the CPU I will not be able to measure the expected peaks in usage." so I have it written to use 75%-88% depending on core count... as when I test on my machine at allowing 100% it locks EVERYTHING until its finished.

 

Thank you!

 

I attached the image from the application for the result I mentioned earlier. I'm currently running a "balanced" Windows power plan so my cores ramp down, but I could change that to "High Performance" to peg my machine for the run and see what happens. Is there any specific scenario you'd appreciate testing in?

 

Edit: Also, not to put too fine a point on it, but those scores are just WAY too close to your 8350 to be meaningful. Literally every score is within a %age or two of the other, and that simply isn't possible if it's being representative of varying architectures etc.

 

Edit2: Looked again, it's less than 0.2% in every case, which is impossible. Something is off with the limiting algorithm perhaps?

 

I'd be willing to test a version of the 100% pegging version you mentioned, if it works out to show a reasonable difference it could be worth looking into.

[FJLJ]

12 minutes ago, Tabs said:

Is there any specific scenario you'd appreciate testing in?

 

Edit: Also, not to put too fine a point on it, but those scores are just WAY too close to your 8350 to be meaningful. Literally every score is within a %age or two of the other, and that simply isn't possible if it's being representative of varying architectures etc.

Really good stuff!  The most helpful tests would be intel intel intel as if you may have noticed I simply dont have a worth while intel cpu to test on.. one thing that may be throwing off results is that with intel cpu's I have noticed that the cpu utilization is able to go over 100%, and is reported as such in windows management, however amd cpus (as far as i can tell) never go over 100%. I had to adjust for this and my sample size is honestly waaay too small to be sure my math is correct. 

 

and... I dunno saying that an intel 6700K @ 4.00 Ghz performs as well as an 8350 at 4.7Ghz may not be too far off. haha  when I run at base clock my result is laughable.. about the same as that i5-7200U @ 2.5Ghz on the results page. Which is probably pretty accurate haha.

 

EDIT: going to go run on my wife's very old laptop.. this should be awesome!

[FJLJ]

Also, if anyone has any ideas for an Icon.. let me know!  haha

[Tabs]

Just now, FJLJ said:

Really good stuff!  The most helpful tests would be intel intel intel as if you may have noticed I simply dont have a worth while intel cpu to test on.. one thing that may be throwing off results is that with intel cpu's I have noticed that the cpu utilization is able to go over 100%, and is reported as such in windows management, however amd cpus (as far as i can tell) never go over 100%. I had to adjust for this and my sample size is honestly waaay too small to be sure my math is correct. 

 

and... I dunno saying that an intel 6700K @ 4.00 Ghz performs as well as an 8350 at 4.7Ghz may not be too far off. haha  when I run at base clock my result is laughable.. about the same as that i5-7200U @ 2.5Ghz on the results page. Which is probably pretty accurate haha.

I've looked into stuff like this before, since it was annoying seeing >100% "Maximum frequency" figures in resource monitor, and from my experiences, "utilisation" is always a static figure that never exceeds 100% - though I have no idea how this is interpreted/exposed programmatically. What goes above 100% is the "maximum frequency" figure - my own machine will show 100% for 4ghz and 110% for 4.4 when only an app is running and pegging turbo. It gets worse for mobile cpu's though - my partners Kaby Lake based machine has a 1.6ghz base clock and a 3.4ghz boost clock, so it goes to stupid values like 180+% when running full pelt and not triggering thermals.

[FJLJ]

Right! I also thought it was only on max freq too. However the only reason I discovered this was I had a friend of mine test on his PC (cant remember now what cpu it has...) and its score made NO SENSE! then I looked at the debug output... at it was adjusting the score for the time over expected.. because although the utilization would have only been ~ 88% of max clock... the system was reporting 110%-115% cpu usage. to combat that I adjust expected by average distance over 100.. this is probably at least in part what is to blame for any result variations. 

[Tabs]

Just now, FJLJ said:

Right! I also thought it was only on max freq too. However the only reason I discovered this was I had a friend of mine test on his PC (cant remember now what cpu it has...) and its score made NO SENSE! then I looked at the debug output... at it was adjusting the score for the time over expected.. because although the utilization would have only been ~ 88% of max clock... the system was reporting 110%-115% cpu usage. to combat that I adjust expected by average distance over 100.. this is probably at least in part what is to blame for any result variations. 

I would love to run the 100% no-limits version of this to see what differences expose themselves in the scores, since it would help to see if it's some limitation in the "normalisation" code is to cause for the crazy scores. Even if frequency differences were to be excluded, my intel cpu and your amd cpu should perform differently in fpu/integer/aes scores (I assume aes score doesn't use acceleration?) so I'm sure something else is at issue there.

[FJLJ]

3 minutes ago, Tabs said:

I would love to run the 100% no-limits version of this to see what differences expose themselves in the scores, since it would help to see if it's some limitation in the "normalisation" code is to cause for the crazy scores. Even if frequency differences were to be excluded, my intel cpu and your amd cpu should perform differently in fpu/integer/aes scores (I assume aes score doesn't use acceleration?) so I'm sure something else is at issue there.

Just got to work for the night. When I get home Ill upload and DM you a link to download the Zero limits version.  thank you for your help!

 

EDIT: correct the AES is just relying on the instruction set included on the cpu.

[WereCat]

i7 4770k

4.6GHz core

4.0GHz cache

 

Left side - Balanced Power Plan

Right Side - Maximum Performance Power Plan

 

[FJLJ]

36 minutes ago, WereCat said:

i7 4770k

4.6GHz core

Thank you very much! Good information here! We will make this as great as it can be!

[LogicalDrm]

 

4770K @ 4.0GHz, 1.15V.

This is inline with @WereCat's results. Though I'd like to see what both of ours look at stock.

 

There's definitely something wrong with how AMD CPUs score. If those are real test results on page, just look at how R7 scores against FX-8350. It just isn't possible to get so similar results with midrange CPU from 2012 and high end CPU from 2017. Intel scores seem to be working, but there would need to be higher sample rate and some other test used as control to see if point difference between CPUs performance is working as supposed.

 

Also bug in mine. Running on Win7 64bit if that helps. Not getting total score.

[FJLJ]

12 hours ago, LoGiCalDrm said:

 

4770K @ 4.0GHz, 1.15V.

This is inline with @WereCat's results. Though I'd like to see what both of ours look at stock.

 

There's definitely something wrong with how AMD CPUs score. If those are real test results on page, just look at how R7 scores against FX-8350. It just isn't possible to get so similar results with midrange CPU from 2012 and high end CPU from 2017. Intel scores seem to be working, but there would need to be higher sample rate and some other test used as control to see if point difference between CPUs performance is working as supposed.

 

Also bug in mine. Running on Win7 64bit if that helps. Not getting total score.

Thank you for the feedback! Looks like it did not properly size after finishing... will test that.

 

Also....

update...

reworking the scoring calculation, I believe I have found an issue in it normalizing a portion completely off.. not good! I too was perplexed by the ryzen score, the only thing I could think of and was confirmed is... that user ran the test while actively using the pc also... in the background... didnt think I would have to include a message to close all running tasks when running...