Relative IPC testing of select Intel CPUs - seeking software suggestions

[porina]

Nearly 5 years ago I did relative IPC testing on Zen 2 vs Skylake, and if you click through, there is also data on Zen, Zen+ and Skylake-X also.

 

I just ordered some Alder Lake bits and intend to repeat something like that testing for various Intel architectures:

Broadwell

Skylake-X (when AVX-512 isn't used, it is generally representative of Skylake, Alder Lake, Coffee Lake, Comet Lake)

Rocket Lake

Alder Lake (P-cores only)

 

Why no AMD? The only recent AMD CPU I currently have is a Zen 3 APU in my laptop, but I don't have sufficient control over it for the intended testing.

 

It should be noted I'm testing the CPU architecture, not the CPU model itself. That's an important distinction, and I will be taking steps to negate the limiting factors that might apply at a model level in order to go towards the architecture ideal performance.

 

The question I have is, what software should I use to test? Following are some at the top of my mind, but more suggestions are welcome. The requirement is that the software runs on Windows, is freely available and CPU focused. No games!

 

• Prime95 - small FFT benchmarks
• Y-cruncher - probably smaller Pi benchmarks, like 25M, maybe 1B. Mainly to try and avoid ram impact.
• Cinebench - At least R15, R23, 2004. I'm not sure R20 adds any value over R23. I'm not sure if there is any point in going older than R15 since that is pre-AVX already.
• Blender
• 3DPM - if I can find it again
• POV-Ray
• 3DMark CPU Profile - not sure about this one

Need more suggestions. That list is too short.

 

Some I wont run:

• Aida64 - it just feels too synthetic and hard to relate to anything tangible. Also I'm too tight to update to the latest version.
• Geekbench - it is its own mix but again, hard to dissect sub results to gain insight
• CPU-Z - just no

 

 

[RONOTHAN##]

6 minutes ago, porina said:

Why no AMD? The only recent AMD CPU I currently have is a Zen 3 APU in my laptop, but I don't have sufficient control over it for the intended testing.

I'm willing to donate 8600G clock cycles if you want to add them. 

 

 

Anyway, for a CPU benchmark, 7-zip compression and decompression is probably a good option to add, though like Y-Cruncher it does have a bit of a memory component to it (no where near as drastic as Y-Cruncher, but large enough that I figured I should mention it). A couple code compile benchmarks (I.E. Firefox, Chromium, Linux Kernel, etc.) might be a good idea as well. V-Ray is also free and might be worth looking at, though I haven't used it personally so don't know quite how good it would be for this system. 

 

9 minutes ago, porina said:

I'm not sure R20 adds any value over R23.

If you aren't using CPUs with AVX-512, not really. R20 and R23 are basically identical aside from the score calculation and AVX-512 support. Sticking to one or the other is probably fine. 

[porina]

1 minute ago, RONOTHAN## said:

I'm willing to donate 8600G clock cycles if you want to add them. 

Can you disable cores and fix clock? If so, we can talk later once I have a better idea of a test methodology for this round. Basically to reduce the impact of caches and memory access, I previously limited the CPU lower cores at low clock. I'm less patient now so I might go up a bit and risk some memory limiting.

 

1 minute ago, RONOTHAN## said:

Anyway, for a CPU benchmark, 7-zip compression and decompression is probably a good option to add

I've used 7-zip since forever but never benchmarked it! I'll have a look later. I kinda suspect this is more a test for the caches and ram though. If so, it wont be a good candidate.

 

1 minute ago, RONOTHAN## said:

though like Y-Cruncher it does have a bit of a memory component to it (no where near as drastic as Y-Cruncher

I think Y-cruncher, when run at small sizes, has a small enough footprint not to be memory limited. 25M is nominally ~200MB so has relatively little data moving compared to say 1B which is ~5GB.

 

1 minute ago, RONOTHAN## said:

A couple code compile benchmarks (I.E. Firefox, Chromium, Linux Kernel, etc.) might be a good idea as well.

I'm concerned there may be too much scope for the environment to vary for this.

 

1 minute ago, RONOTHAN## said:

V-Ray is also free and might be worth looking at, though I haven't used it personally so don't know quite how good it would be for this system. 

Thanks, looks like a possibility. I'll have to investigate further.

 

1 minute ago, RONOTHAN## said:

If you aren't using CPUs with AVX-512, not really. R20 and R23 are basically identical aside from the score calculation and AVX-512 support. Sticking to one or the other is probably fine. 

R15 was before they used AVX. To my understanding R20 and R23 can use AVX, but doesn't use AVX-512. 

 

Quote

AVX-512 is used in Cinebench 2024, but in such low amounts that it’s irrelevant.

https://chipsandcheese.com/2023/10/22/cinebench-2024-reviewing-the-benchmark/

[Electronics Wizardy]

How about video encoding using something like ffmpeg and svt-av1 for encoding.

 

 

I'd also look at encryption performance. There is often acceleration for that on cpus so take that into account.

[RONOTHAN##]

56 minutes ago, porina said:

Can you disable cores and fix clock? If so, we can talk later once I have a better idea of a test methodology for this round. Basically to reduce the impact of caches and memory access, I previously limited the CPU lower cores at low clock. I'm less patient now so I might go up a bit and risk some memory limiting.

Yeah, I have full control over which cores are disabled and what clock speeds they run at, just checked by running it as a triple core locked at 2GHz. I also have the ability to enable/disable hyperthreading if you end up caring about that. 

 

56 minutes ago, porina said:

I've used 7-zip since forever but never benchmarked it! I'll have a look later. I kinda suspect this is more a test for the caches and ram though. If so, it wont be a good candidate.

 

1 hour ago, RONOTHAN## said:

though like Y-Cruncher it does have a bit of a memory component to it (no where near as drastic as Y-Cruncher

I think Y-cruncher, when run at small sizes, has a small enough footprint not to be memory limited. 25M is nominally ~200MB so has relatively little data moving compared to say 1B which is ~5GB.

From my testing of it at least, it scales linearly with core clock similarly to how Y-Cruncher does. It just also scales scales quite a bit with memory performance like Y-Cruncher 1B does, but it more scales from timings rather than clock speed like how Y-Cruncher does. 

Spoiler

 

56 minutes ago, porina said:

To my understanding R20 and R23 can use AVX, but doesn't use AVX-512. 

AFAIK R20 does have AVX-512 support, but then it was removed for R23 due to it having little benefit. 

[porina]

8 minutes ago, RONOTHAN## said:

Yeah, I have full control over which cores are disabled and what clock speeds they run at, just checked by running it as a triple core locked at 2GHz. I also have the ability to enable/disable hyperthreading if you end up caring about that. 

It will be interesting to see HT scaling too, but obviously that doubles the testing.

 

I'll decide on the exact configuration once I get Alder Lake set up since that'll probably be the fastest of what I have, and I can see how ram affects it. Zen 4 should be similar or faster I'd guess.

 

8 minutes ago, RONOTHAN## said:

From my testing of it at least, it scales linearly with core clock similarly to how Y-Cruncher does. It just also scales scales quite a bit with memory performance like Y-Cruncher 1B does, but it more scales from timings rather than clock speed like how Y-Cruncher does. 

That would make it less ideal as a benchmark for this use since I'm focusing on the execution and not ram settings. Still, I'll try it with reduced compute settings and see if that makes the ram influence small enough.

 

8 minutes ago, RONOTHAN## said:

AFAIK R20 does have AVX-512 support, but then it was removed for R23 due to it having little benefit. 

Interesting. The reference I dug up only references R23, so I'd leave R20 as an unknown state. Well, I could try it on my Skylake-X systems with and without AVX-512 enabled. Then again, the later reference stating that 2024's usage of AVX-512 is "irrelevant" might roll back to the older versions too.

[RONOTHAN##]

17 minutes ago, porina said:

That would make it less ideal as a benchmark for this use since I'm focusing on the execution and not ram settings. Still, I'll try it with reduced compute settings and see if that makes the ram influence small enough.

I mean, if you look at the benchmarks I threw in there, the difference in score between 6000 CL30 and 5200 JEDEC was only about 2% (~80800 vs. ~79000). It's definitely measurable, but it's not big enough it should matter too much. On Raptor Lake (13700k) it's about 6% from 7200 CL34 to 4800 CL40 (161k vs. 151k), so more noticeable, but still probably small enough to be OK. 

[porina]

8 minutes ago, RONOTHAN## said:

I mean, if you look at the benchmarks I through in there, the difference in score between 6000 CL30 and 5200 JEDEC was only about 2% (~80800 vs. ~79000). It's definitely measurable, but it's not big enough it should matter too much.

I missed that. Ok, not nearly as bad as it sounded. With reduced CPU resources, that difference should drop further.

[freeagent]

I like Linpack Xtreme. GFlops tell the tale

 

I usually run the 10GB load. It does hit a bit harder than prime does in my opinion.

[porina]

5 minutes ago, freeagent said:

I like Linpack Xtreme. GFlops tell the tale

 

I usually run the 10GB load. It does hit a bit harder than prime does in my opinion.

Will have to play about with it. In general I consider Prime95 and Linpack in the same load class, as both throw FP64 around hard. Does it use AVX-512? It says it uses MKL from 2018, and Skylake-X was released 2017 so it could.

[freeagent]

51 minutes ago, porina said:

Does it use AVX-512?

I actually forgot the answer to that. The author mentioned it once, but I do not recall what "it" was

[porina]

Done some pre-testing just on the 12100F. To have a consistent clock I turned turbo off, so it ran at base clock of 3.3 GHz. I also tried running it with and without HT, as well as observing the power consumption. With that I hope to get some insight on how much perf HT could bring, at what power cost. When introduced it was claimed to give a bigger perf increase than power draw increase. Let's see if that happens.

 

Note the 12100F system I running has a single channel dual rank 3200 ram module as I can't physically fit a 2nd due to cooler getting in the way. I have not checked if any of these tests may be limited from this.

 

Cinebench R15: HT gave a 40% increase in score for a 19% increase in power. The perf increase is bigger than I expected, given older CPUs were around the 30% mark.

Y-cruncher: 13% increase from HT at both 25m and 1b. I saw 14% increase in power 1b. The 25m test was too short for me to look at the power.

POV-Ray: 18% increase in perf for 17% increase in power.

Linpack Xtreme: 1.5% increase in throughput for no change in power. Margin of error?

Prime95 128k throughput: No change in perf for 11% increase in power draw. This workload is known not to benefit from HT, and the overhead of it even being there works against it.

7-Zip: 17% faster compression, 47% faster decompression, 16% more power draw.

 

Results are all over the place. Cinebench R15 and 7-Zip decompression see a great benefit from HT, and improves perf/W. POV-Ray, Y-Cruncher and 7-Zip compression see an improvement in performance but no significant change in perf/W. Linpack Xtreme and Prime95 both don't see a change in perf, with P95 seeing a big drop to perf/W.

 

 

Actual power usage of each workload (rounded) with HT on:

Cinebench R15: 35W

Y-cruncher: 40W

POV-Ray: 35W

Linpack Xtreme: 37W

7-Zip: 29W

Prime95: 45W (41W without HT)

 

This is why I refer to Cinebench R15 as a "light" workload, and Prime95 to be a "heavy" one. If instead you run a fixed power limit instead of fixed clock, then the heavier workloads will end up with the CPU at a lower clock than light workloads.

 

 

Some other observations on software for the purposes for future testing:

I didn't run Cinebench 2024 because I forgot how SLOW it runs. I probably should include it but I didn't have the patience to do it in this set. R23 doesn't seem to be easily available stand alone and I didn't want to install BenchMate to get it for now.

Blender benchmark launcher wouldn't actually download the benchmarks so they're out.

I was going to use HWBOT x265 as a video encoder test, but it refuses to run without HPET, and I'm not enabling it just for that.