AMD says these are the same... We DISAGREE... - Testing 12 of the Same CPU Video and Process Doc

[AAVVIronAlex]

On 1/26/2024 at 12:04 AM, AdamFromLTT said:

 

 

Do you really know if your CPU is performing the same as the ones we review? We don’t know. But we know that if we want to increase our testing capacity, we need to PARALLELIZE. But that means we need nearly identical test benches. And trying to make that happen sent us down a far deeper rabbit hole than we could have anticipated.

--------------------------------------------
“Testing 12 of the Same CPUs”

 

AMD CPU Variance Testing Documentation

 

Testing Dates: 2023-06-04 to 2023-08-01

AMD-CPUVAR-2023

For testing variance of the CPUs, all hardware will remain the same on the bench. The same motherboard, graphics card, SSD and other system components need to be used for all testing across all chips to ensure that the only variable is the CPU itself. We verify that the fans for the Noctua cooler are installed in the same spot every time. Every time we change a CPU, we ensure that we go into the BIOS, set the BIOS settings to default, save and restart, then go in and set the BIOS settings to our standard configuration, EXPO on, Re-Bar on, IGPU disabled, fan curves are set appropriately and then save and restart again.

 

Picture 1: Test Bench hardware configuration

 

After testing all of the CPUs, we use one as the “control” CPU to test the variance of the 3 motherboards and 3 sets of RAM. When testing the motherboards, we utilize the same RAM used in the CPU variance testing. When testing the RAM sets, we utilize the motherboard that was used in the CPU variance testing. This is to see what (if any) variance there will be between the 3 benches. Again, for every hardware change, we repeat the same BIOS adjustments as above.

 

All testing should be done within the environmental chamber at 20°C with the test bench oriented in the same direction for purposes of airflow. There will be taped markings on where to align the test bench to ensure that it is located in the same spot after any hardware changes.

 

Picture 2: PTM7950 pad orientation and size used for test reference.

 

For testing, we will be using PTM7950 pads cut to the same size for each CPU to take out any guesswork about thermal paste applications. When using the PTM pad make sure to heat soak it before testing.

To help assist with less run-to-run variance, we will be using Bill2’s Process Manager to run all applications that are not games at High Priority (not Real-Time as that can crash the system in some instances). The profiles for each application exe will need to be set up in the Process Manager to ensure that they run in High Priority every time during the testing. We verify that these profiles are all working correctly as intended before collecting data for the project. This is especially true of Cinebench R23 as it actually sets itself to Below Normal Priority by default, and if we wait for the program to finish initializing on open before changing the priority, it will again try to set itself back to Below Normal Priority.

 

To start setting up the system we will need the following hardware:

 

Graphics Card: NVIDIA RTX 4090 FE (Asset Tag: C2940)

Storage: Samsung 980 Pro 2TB (Asset Tag: C5765)

Cooling: Noctua NH-D15 + 2x NF-A15 PWM fans (Asset Tag: C3382)

Thermal Interface Material: PTM7950

Motherboard: Gigabyte X670E Aorus Xtreme (Asset Tag: C3511)

Memory: 2X16GB G.Skill Trident Z5 Neo 6000 MT/s CL30-38-38-96 (SN 20032 & 20031)

Power Supply: MSI MEG Ai1300P (Asset Tag: C2947)
CPU: 12x Ryzen 9 7800X3D (Asset Tags: C4782, C5281, C5371, C5383, C5424, C5504, C5530, C5540, C5546, C5643, C5660, C5715)
 

Everything will be installed on the same Open Benchtable V2 for portability. We start by installing the power supply, motherboard and SSD on the bench. We will start testing with the C4782 CPU, so install that CPU into the socket. Finish building the system (firstly with just regular thermal paste as we will be installing the PTM after system configuration has been completed), update the BIOS to version F9a, set the BIOS settings appropriately to our standard configuration, and then install the custom Windows image used for all our test benches.

Once the image has finished installing, we install all the latest drivers for the hardware from AMD’s site for the chipset and from Gigabyte’s site for any others that did not get installed correctly. We go through installing all required programs and games used for this project. Some are self-contained in Markbench but we will be testing the following:

 

 

 

After installing all of the applications and games, we obtain all the current Windows updates available, then once all the updates have been installed we pause the Windows updates for 5 weeks to ensure that there are no changes to the operating system.

We install Bill2’s Process Manager as mentioned above and set profiles for all the exe’s that perform the benchmark (some applications launch a separate exe that contains the benchmark’s load). If unsure what the exe is called, we run the benchmark and check it in the task manager to see which exe is running and where it is located. Once this has been set up, clear all temp files in the %temp% folder, optimize the SSD, set the recycling bin to 15MB, and then shut down the system so we can prepare the CPU for its testing. 

 

CPU VARIANCE TESTING

We remove the NH-D15, clean all thermal paste off the CPU and heatsink with Isopropyl alcohol to ensure that the surfaces are clean of any residue, and then we install the PTM pad.

Install the PTM by first peeling one of the plastic protective films, and aligning the PTM as close to the image above as possible. Ensure you squeeze out any air bubbles when installing it onto the CPU before removing the second protective plastic film. When installing the PTM pad, removing the plastic film is easiest with a set of tweezers and a guitar pick or spudger to make sure that the pad doesn’t try to also peel back off the CPU’s surface when removing the plastic protective film. 

 

After installing the PTM pad, we install the Noctua NH-D15 with the orientation so that you can read the Noctua logo in the same orientation as the Aorus Xtreme chipset heatsink’s wording, making sure to torque down the screws on the heatsink till they stop. 

Picture 3: Noctua NH-D15 orientation

 

Once the heatsink and fans have been reinstalled on the system boot the system back up into the BIOS. As iterated above, we set the BIOS settings to default, save and restart, then go in and set the BIOS settings to our standard configuration, EXPO on, Re-Bar on, IGPU disabled, fan curves set appropriately and then save and restart again. Run OCCT for 30 minutes to let the pad flow as it should, shut off the system for 30 minutes to let the PTM cool, then turn on the system and begin testing.

 

When running non-game applications for benchmarking, we ensure that all other applications such as Steam are closed in the tray, keeping only Bill2’s Process Manager open, utilizing MarkBench for any of our automated non-game benchmarks and closing it for any that do not use it. Steam updates can come in at any time and cause the run to become invalid so it must remain closed for any non-game tests. On the flip side, when running the game benchmarks, ensure that there are no other application windows open on the system or anything else running in the tray aside from Steam and MarkBench to ensure its run-to-run consistency.
 

For all MarkBench tests, we ensure that we tag the sessions with the appropriate asset tag for traceability. For any screenshots, label the screenshot with the appropriate application name, run number and asset tag code for the CPU.

 

After any major test (such as SpecWS), we restart the system to flush the RAM to ensure consistency. Every time we restart the system, we make sure to wait for about 5 minutes to let the OS do any required system load checks. 

 

Once all tests have been completed, before removing the CPU, we ensure that the runs are all valid. Once the CPU has been removed, if it is required to reinstall back into the system we will have to redo ALL tests as we have effectively changed the “configuration”.

 

After finishing testing the first CPU and verifying all the runs are valid, we shut down the system, and let it cool for approximately 15 minutes to allow the PTM to turn back to a solid state. This helps with the cleaning process as it is much easier to simply scrape most of the hardened PTM off the CPU rather than deal with it as a liquid or paste. We remove the heatsink from the CPU, scrape all the PTM from the heatsink and CPU using a plastic spudger or guitar pick, then clean both surfaces again with isopropyl alcohol. After cleaning them, remove the CPU, and repeat the process with the next CPU. After installing a new CPU, we reinstall the chipset drivers to verify that the X3D’s V-Cache is running optimally before running the tests followed by a restart.

 

RAM VARIANCE TESTING

Once all CPU tests have been completed, with the last CPU installed still in the motherboard, we will use it to check any variance of the RAM next. In a similar process to the CPU variance testing, we will be simply swapping the RAM kits on the motherboard while leaving the same CPU in the system (C5715). This should be easier to control as we are simply removing the one fan from the Noctua cooler to get access to the RAM slots.

The RAM kits we will need to test variance vs our current set used to test the CPUs are: SN ending 20034 & 20033 and SN ending 73909 & 73910 (Asset Tags: C5914 & C5915). 

We remove the first set of RAM used to test the CPU variance and make sure to keep note of which set it is and what order it was installed in (SN ending in 20031 & 20032) as we will be using that set for the motherboard variance later. Then install the next set of RAM and repeat all the tests listed above. Similar to the CPU, reset the BIOS again and readjust the BIOS settings appropriately. This will help ensure that the RAM is running optimally.

 

We tag the sessions with the CPU asset tag as well as the RAM used for any MarkBench tests and for any screenshots, we label them with the appropriate application name, run number, CPU asset tag, and RAM serial number digits (last 5 digits) for traceability purposes.

 

Before removing the RAM, all runs are validated (similar to the CPU testing) as we want to leave the system configured as is to help control any potential variable of even the RAM being installed in the opposite slots.

 

MOTHERBOARD VARIANCE TESTING

Once all RAM tests have been completed, we will now be checking for any motherboard variance between the three we will be using for our benches using the CPU (C5715) and RAM (SN 20032 & 20031) from the CPU variance testing. This will be the one where installation differences could play more of a factor. However, by using the same cooler, CPU, and RAM (ensuring that they are always installed in the same order for the slots) and using PTM pads should limit any installation variance issues.

The motherboards we will need to test variance vs the one used for the CPU testing are: C2941 and C5378.

 

We will have to tear down most of the system to test the motherboard swaps. Don’t forget to take out the SSD from the system! We pay close attention to the orientation of the heatsink and its motherboard mounts, the RAM order and that the SSD is installed in the same slot.

Once all the other system components have been installed including the PTM, we boot up the system. We verify in the BIOS that the motherboard is running version F9a, if not flash it. Similar to the CPU, reset the BIOS again and readjust the BIOS settings appropriately. This will help ensure that the system is configured the same as the other motherboard.

 

We tag the sessions with the CPU asset tag and the appropriate motherboard asset tag for any MarkBench tests. For any screenshots, label the screenshots with the appropriate application name, run number, CPU asset tag, and motherboard asset tag for traceability purposes.

 

Before removing all of the hardware from the system, and verifying all runs are valid (similar to the CPU testing) we want to leave the system configured as is to help control any potential variable of hardware being installed ever so slightly differently.

 

–

End of Documentation
 

I still do not understand how we can accept these things turboing to until they reach 95C.

[cmndr]

58 minutes ago, AAVVIronAlex said:

I still do not understand how we can accept these things turboing to until they reach 95C.

We have 15 years of data on laptops turboing to 95C and not dying. What's even worse for laptops is that they often have big temperature swings which is really rough on things. Hitting 95C is not a big deal, especially if you're not cranking the voltage. 

Something else is going to die before the CPU does (e.g. the motherboard) and by the time the CPU does die you can buy something similar on ebay for cheap. 
 

[AAVVIronAlex]

14 hours ago, cmndr said:

We have 15 years of data on laptops turboing to 95C and not dying. What's even worse for laptops is that they often have big temperature swings which is really rough on things. Hitting 95C is not a big deal, especially if you're not cranking the voltage. 

Something else is going to die before the CPU does (e.g. the motherboard) and by the time the CPU does die you can buy something similar on ebay for cheap. 
 

It is still inconvenient, not everything can cool that. My brother has a 7600X and it just goes, like it does not care if it is under a NH-D15 or what, it just ups the clocks. We ended up lowering them to get a stable 5.3GHz with 75 as the maximum temperature.

[kokosnh]

20 hours ago, LMGcommunity said:

We still want our test results to reflect the average consumer's experience.

That’s not he’s point. 

looking for consistent test rigs =/= benchmarking, 

you wanted to test the GPU and RAM, and mobo for consistency, so lock the other components, and unlock only the one, you look for differences.

and repeat that

 

And when you have the most similar test rigs for the rest of the system, you can then unlock it and test the CPU ( but I would still lock the ram timings).

 

you are introducing variable, where you don't have to, this is just flawed from the concept. 
looking for consistent test rigs, is not benchmarking, you do not want it to be close to real life, you want it close to lab environment, where everything is controlled.

[GarlicDeliverySystem]

On 1/30/2024 at 3:24 PM, kokosnh said:

That’s not he’s point. 

looking for consistent test rigs =/= benchmarking, 

I would agree with this, but you have basically two questions intermingled in this video:

• finding/defining consistent benches for GPU tests, isolating/fixing as many variables as possible
• understanding the variance a consumer can experience with reasonable similar setups.

In the first sense, I agree: locking everything to the lowest common denominator would probably give you the most consistent test platform. It would just happen to underperform compared to other, similar setups.

 

However, for the viewer the question is more: how close is my rig of X CPU, Y MOBO, Z RAM to someone else buying the same XYZ components and doing the same recommended high level changes (i.e. enabling EXPO/DOCP etc.). From that perspective I can understand if they don't check the secondary timings, since those should be related to the CPU memory controller (if the RAM is always the same sticks).

[kokosnh]

3 hours ago, GarlicDeliverySystem said:

However, for the viewer the question is more: how close is my rig of X CPU, Y MOBO, Z RAM to someone else buying the same XYZ components and doing the same recommended high level changes (i.e. enabling EXPO/DOCP etc.). From that perspective I can understand if they don't check the secondary timings, since those should be related to the CPU memory controller (if the RAM is always the same sticks).

And they can still do that, when benchmarking… there’s no telling they have to benchmark only the most similar one, you can still test the others and present the results.

 

the problem with for example secondary timings on RAM is that, they can literally change for boot to boot.  nobody is asking to lower the ram performance, or change it, just look what secondary timings ram have, and lock it.

 

the same with for example CPU voltage, locking it little lower, can actually increase performance, as cpu wouldn't throttle so hard.
you can still try to lock it, with relatively similar performance, and that’s what the goal should be.

[GarlicDeliverySystem]

47 minutes ago, kokosnh said:

the problem with for example secondary timings on RAM is that, they can literally change for boot to boot.  nobody is asking to lower the ram performance, or change it, just look what secondary timings ram have, and lock it.

 

the same with for example CPU voltage, locking it little lower, can actually increase performance, as cpu wouldn't throttle so hard.
you can still try to lock it, with relatively similar performance, and that’s what the goal should be.

Yes, for the testbench use case. My point of locking everything at the lowest common denominator was mostly about convenience for the testing there, as you can save a lot of headache trying to push some underachiever CPUs to the tighter specs of the benches.

 

However my point still stands that for a consumer perspective this doesn't matter. The typical consumer, even the more knowledgeable ones who watch these (and others') videos won't necessarily go in their BIOS to tighten/fix all the timings and fiddle with voltages that much. They leave this up to the CPU and motherboard manufacturers to deal with, so testing how much variance is in there would be highly relevant to consumers imho.

Imagine having an error bar for that and not for the actual measurement itself, like knowing that i.e. certain CPUs have tighter specs while others are much more of a lottery. That could affect someone's buying decision.

 

Now pair the two, where you then actually look at the variances encountered in those auto-timings and how they affect performance. Guess overclockers would be highly interested in data like this for every major CPU out there.

[kokosnh]

@GarlicDeliverySystemand I agree with you, but we are talking about different things.
the point he touch was looking for similar performance components.
the benchmarking can be still done with some minor improvements. 
Those are separate things…

we are not talking about benchmarking! 
I did respond to your statement, about the idea of locking components, but that was just an example. The point never was about benchmarking,

 

[powertoys]

On 1/29/2024 at 5:04 PM, cmndr said:

We have 15 years of data on laptops turboing to 95C and not dying. What's even worse for laptops is that they often have big temperature swings which is really rough on things. Hitting 95C is not a big deal, especially if you're not cranking the voltage. 

Something else is going to die before the CPU does (e.g. the motherboard) and by the time the CPU does die you can buy something similar on ebay for cheap. 
 

 

It's almost as if Intel / AMD's engineers know what the heck they are doing.

[Ksog]

I would love to see the result numbers in table mode, so we can find out where our 7800X3D CPU are in the spectrum