Upgrading my PC for improved Inference

[SG-O]

Budget (including currency): Under 6k Euro (can be more with a very good reason)

Country: Germany

Games, programs or workloads that it will be used for: While the upgrade is primarily intended to speed up AI inference on CPU, compiling large codebases and improving things like shader compilation in Unreal, I would still like to have a competent gaming system.

Other details I only want to upgrade my CPU, RAM, Motherboard and maybe PSU.

At the moment I have an I7-8700k, ROG Maximus X Code, 2x 8GB DDR4-3866. My PSU is a Corsair HX750.

 

My considerations, research and planning is documented here:

https://docs.google.com/spreadsheets/d/1LSCbwWWkWEeDxZxI6IBFOS0Fw0J04e71bYHYbnFFWPE/edit?usp=drive_link

 

What I want is a lot more RAM (min. 256GB, maybe even 512), a faster CPU with support for AVX 512, maybe AMX. 

 

I don't intend to upgrade my GPU at this moment as GPUs with the amount of RAM i would like to have, don't exist with a reasonable price tag, yet.

 

My current preference is the Threadripper 7000 series but an Intel based system would also be acceptable.

 

If I go Threadripper, I don't know if the 8 RAM channels of the PRO lineup would be advantageous for my use case and improve the systems speed.

 

My cooling might need upgrading as well. I have a NH-D15 at the moment but I think that a custom water-cooling loop is the way to go here. My case is a Corsair 750D that I absolutely want to keep.

 

One last thing: I run Linux Mint as my only OS and virtualization support is a must have feature.

[Tetras]

Where do you go to find benchmarks for the software you use? TPU include AI/ML in their reviews here, but I don't know if it is applicable at all to what you're doing. They also have compile and UE5 benchmarks.

 

 

[brob]

There is not much choice in the desktop segment if one needs more than 192GB of memory. Motherboards with the TRX50 chipset offer up to 1TB of 4-channel memory.

 

I don't believe there is an NH-D15 mounting adapter for the sTR5 socket. Not really surprising given the CPU have a 350W+ TDP.

 

If you want gaming, get a separate gaming system. 

[PDifolco]

1 hour ago, SG-O said:

Budget (including currency): Under 6k Euro (can be more with a very good reason)

Country: Germany

Games, programs or workloads that it will be used for: While the upgrade is primarily intended to speed up AI inference on CPU, compiling large codebases and improving things like shader compilation in Unreal, I would still like to have a competent gaming system.

Other details I only want to upgrade my CPU, RAM, Motherboard and maybe PSU.

At the moment I have an I7-8700k, ROG Maximus X Code, 2x 8GB DDR4-3866. My PSU is a Corsair HX750.

 

What I want is a lot more RAM (min. 256GB, maybe even 512), a faster CPU with support for AVX 512, maybe AMX. 

 

I don't intend to upgrade my GPU at this moment as GPUs with the amount of RAM i would like to have, don't exist with a reasonable price tag, yet.

 

My current preference is the Threadripper 7000 series but an Intel based system would also be acceptable.

 

If I go Threadripper, I don't know if the 8 RAM channels of the PRO lineup would be advantageous for my use case and improve the systems speed.

 

My cooling might need upgrading as well. I have a NH-D15 at the moment but I think that a custom water-cooling loop is the way to go here. My case is a Corsair 750D that I absolutely want to keep.

 

One last thing: I run Linux Mint as my only OS and virtualization support is a must have feature.

From what I've gathered, 7000 TR PRO are super expensive and designed for server task

The just announced HEDT TR (non PRO) 7000 to be released in Nov23 could be better for you, the "midrange" chip (7970X) 32C/64T will be $2.5K and may be what you need

The "top" model 64C/128T will be $5K so over your budget...

Regarding cooling not sure what can cool 350W, but I won't put a watercooling in a production PC if things run overnight or such

 

 

[Gorgon]

1 hour ago, SG-O said:

...

My current preference is the Threadripper 7000 series but an Intel based system would also be acceptable.

 

If I go Threadripper, I don't know if the 8 RAM channels of the PRO lineup would be advantageous for my use case and improve the systems speed.

 

My cooling might need upgrading as well. I have a NH-D15 at the moment but I think that a custom water-cooling loop is the way to go here. My case is a Corsair 750D that I absolutely want to keep.

 

One last thing: I run Linux Mint as my only OS and virtualization support is a must have feature.

I would wait for Wendel to test socket TRx50 motherboards and non-Pro 7000-series CPUs. The non-pro motherboards should have no problem supporting huge amounts of memory on just 4 sockets as they only support RDIMMs so no plain DDR5 or UDIMM support but remember that DDR5 is sort of dual banked already. Your NH-D15 might work but will, likely require offset mounting which will depend on where the chiplets are on the specific processor but a TRx50 specific cooler will work much better and handle more of the the 350W load.

 

[leclod]

My 12400 cpu supports AVX512 and 128GB Ram (I believe most 12/13/14 cpus can't avx512 because of the E cores)

(I have not much understanding about the requirements, just thought I'd get that out there.)

Edited October 20, 2023 by leclod

[SG-O]

1 hour ago, PDifolco said:

From what I've gathered, 7000 TR PRO are super expensive and designed for server task

The just announced HEDT TR (non PRO) 7000 to be released in Nov23 could be better for you, the "midrange" chip (7970X) 32C/64T will be $2.5K and may be what you need

The "top" model 64C/128T will be $5K so over your budget...

Regarding cooling not sure what can cool 350W, but I won't put a watercooling in a production PC if things run overnight or such

 

 

Thanks for your suggestions. From what I heard in the news coverage about the Threadripper launch, no benchmarks for the HEDT parts are available, yet. I've only seen numbers for the 96 core Pro variant. So I will definitely wait for more information when it comes out. This is just to get some preliminary feedback about my plans and find out if i missed something in my research up to this point.

 

My biggest question is about the effects the 4 additional Ram channels would have on working with very large datasets. But this probably can't be answered until the benchmarks are available.

 

The NH-D15 is probably to small to properly cover the heatspreader. I will likely need something else. If I go the water-cooling route I will definitely use something like a Leakshield.

[Gorgon]

2 hours ago, SG-O said:

Thanks for your suggestions. From what I heard in the news coverage about the Threadripper launch, no benchmarks for the HEDT parts are available, yet. I've only seen numbers for the 96 core Pro variant. So I will definitely wait for more information when it comes out. This is just to get some preliminary feedback about my plans and find out if i missed something in my research up to this point.

 

My biggest question is about the effects the 4 additional Ram channels would have on working with very large datasets. But this probably can't be answered until the benchmarks are available.

 

The NH-D15 is probably to small to properly cover the heatspreader. I will likely need something else. If I go the water-cooling route I will definitely use something like a Leakshield.

This just out from Noctua on the new CPUs

[SG-O]

12 minutes ago, Gorgon said:

This just out from Noctua on the new CPUs

Pretty much confirmes my suspicions that the NH-D15 is the wrong cooler for the new Threadrippers. If I have to get a new cooling setup, I might as well go water-cooling with the huge amount heat the new CPUs produce.

[Gorgon]

4 hours ago, SG-O said:

Pretty much confirmes my suspicions that the NH-D15 is the wrong cooler for the new Threadrippers. If I have to get a new cooling setup, I might as well go water-cooling with the huge amount heat the new CPUs produce.

The problem is finding a good AIO cooler for Threadripper has been challenging. The vast majority of AIO coolers are based on older designs and their coldplate is way smaller than the IHS of the Threadripper CPUs. Alpha Cool have AIOs specifically for Threadrippers, but you'll have to wait for a mounting kit upgrade for the new socket. Custom loop is the way to go if you must squeeze the most performance out of it.

 

Do note that Wendel uses mainly Noctua Air Coolers on his Threadripper Systems, yes you may not get the full performance, but they will last forever and NOT leak over your very expensive system.

[SG-O]

43 minutes ago, Gorgon said:

The problem is finding a good AIO cooler for Threadripper has been challenging. The vast majority of AIO coolers are based on older designs and their coldplate is way smaller than the IHS of the Threadripper CPUs. Alpha Cool have AIOs specifically for Threadrippers, but you'll have to wait for a mounting kit upgrade for the new socket. Custom loop is the way to go if you must squeeze the most performance out of it.

 

Do note that Wendel uses mainly Noctua Air Coolers on his Threadripper Systems, yes you may not get the full performance, but they will last forever and NOT leak over your very expensive system.

I love my noctua! But if I'm going to spend that much money on a new CPU + MB + Ram, I want to get all it's performance.

 

I'm considering building a custom loop for this system. I always wanted to build one and this is the perfect opportunity. And to protect from leaks im going to use a Leakshield.

[igormp]

Phoronix has some benchmarks on CPU inference:

https://openbenchmarking.org/suite/pts/machine-learning

https://openbenchmarking.org/test/pts/tensorflow&eval=710d858e39b9394d3d7af3968973103c64ef880e#metrics

 

Just pick a case that's similar to yours and you should have a pretty nice idea of what to look for.

 

On 10/20/2023 at 9:32 AM, SG-O said:

What I want is a lot more RAM (min. 256GB, maybe even 512), a faster CPU with support for AVX 512, maybe AMX. 

 

So, either the new TRs or the SR Xeon WS.

On 10/20/2023 at 9:32 AM, SG-O said:

If I go Threadripper, I don't know if the 8 RAM channels of the PRO lineup would be advantageous for my use case and improve the systems speed.

 

It really depends on what kind of dataset and model you're working on. It may make a ton of difference, or no difference at all.

On 10/20/2023 at 11:49 AM, SG-O said:

So I will definitely wait for more information when it comes out. This is just to get some preliminary feedback about my plans and find out if i missed something in my research up to this point.

Do keep an eye on phoronix for their tests. Depending on your model/datset/framework, Intel may have a lead with their AVX512 implementation + accelerators built into the CPU.

On 10/20/2023 at 11:49 AM, SG-O said:

My biggest question is about the effects the 4 additional Ram channels would have on working with very large datasets. But this probably can't be answered until the benchmarks are available.

 

The access patterns of your dataset during your inferences are more relevant than the dataset size itself tbh, could you give some details on that?

 

[SG-O]

On 11/1/2023 at 5:20 AM, igormp said:

Phoronix has some benchmarks on CPU inference:

https://openbenchmarking.org/suite/pts/machine-learning

https://openbenchmarking.org/test/pts/tensorflow&eval=710d858e39b9394d3d7af3968973103c64ef880e#metrics

 

Just pick a case that's similar to yours and you should have a pretty nice idea of what to look for.

 

So, either the new TRs or the SR Xeon WS.

It really depends on what kind of dataset and model you're working on. It may make a ton of difference, or no difference at all.

Do keep an eye on phoronix for their tests. Depending on your model/datset/framework, Intel may have a lead with their AVX512 implementation + accelerators built into the CPU.

The access patterns of your dataset during your inferences are more relevant than the dataset size itself tbh, could you give some details on that?

 

Thanks for these resources. I hope that there will be some CPU centric results for the new TR soon.

 

You are right that answers to many of my questions are highly dependent on my specific use cases, which I have forgotten to specify. I'm primarily interested in running LLAMA based LLMs (including 70b+), fairseq, etc offline as well as image generation models like sdxl.

 

Please don't forget that I don't want this to be a purely work oriented build. The CPU has to be capable of running the latest games decently.

[igormp]

1 hour ago, SG-O said:

You are right that answers to many of my questions are highly dependent on my specific use cases, which I have forgotten to specify. I'm primarily interested in running LLAMA based LLMs (including 70b+), fairseq, etc offline as well as image generation models like sdxl.

Yeah, for stuff like LLAMA and other LLMs memory bandwidth does make tons of difference, so going for more channels will be better.

Image generation not so much.

 

1 hour ago, SG-O said:

Please don't forget that I don't want this to be a purely work oriented build. The CPU has to be capable of running the latest games decently.

You didn't mention what GPU you have, nor which games, resolution and target framerate you want. Anyhow, any of those modern WS CPUs should do the job well enough as long as you are not targeting 200hz+ in competitive games.

[SG-O]

14 hours ago, igormp said:

You didn't mention what GPU you have, nor which games, resolution and target framerate you want. Anyhow, any of those modern WS CPUs should do the job well enough as long as you are not targeting 200hz+ in competitive games.

I still have an old GTX 1080 which I plan on upgrading once gaming class GPUs with 48GB+ VRAM are available. I just don't want my choice of CPU today to ruin my gaming performance when I upgrade.

And no, I'm not a competitive gamer.

[igormp]

1 hour ago, SG-O said:

I still have an old GTX 1080 which I plan on upgrading once gaming class GPUs with 48GB+ VRAM are available. I just don't want my choice of CPU today to ruin my gaming performance when I upgrade.

And no, I'm not a competitive gamer.

Such GPUs aren't going to be a thing in the near future, so I guess you'll wait for quite some time.

Why not just go for 2x3090 or 2x4090?

 

Anyhow, those modern WS CPUs are planty fast and won't hold you back, or will hold you back a lot by the time you get such a GPU, but any current consumer CPU is likely to do the same given the age it'll have by then.

[SG-O]

Now that the CPUs supposedly have been officially launched I have started to work on a few more detailed builds. I quickly realized, that all of the Motherboards that have been announced aren't available, yet. So I had the guess a price point.

No reviews of the Pro parts comparing them to the HEDT ones are available yet.

 

Also the RAM availability of 64GB R-Dimm with more than 4800MT/s is almost non existent. After a lot of searching I found 64GB sticks from SK Hynix with 5600 Cl46 (HMCG94AGBRA181N). Do any of you know of a better choice?

 

Another thing to find out is the support for sTR5 by the water block.

 

Here is my current research/plan:

https://docs.google.com/spreadsheets/d/1LSCbwWWkWEeDxZxI6IBFOS0Fw0J04e71bYHYbnFFWPE/edit?usp=drivesdk

 

All of this means that I will wait for a month or two until all components are released, more reviews are available and hopefully better ram is available.

[igormp]

17 minutes ago, SG-O said:

No reviews of the Pro parts comparing them to the HEDT ones are available yet.

This one may be your best figure:

https://www.phoronix.com/review/amd-linux-threadripper-pro-7995wx

18 minutes ago, SG-O said:

Also the RAM availability of 64GB R-Dimm with more than 4800MT/s is almost non existent. After a lot of searching I found 64GB sticks from SK Hynix with 5600 Cl46 (HMCG94AGBRA181N). Do any of you know of a better choice?

Yeah, you'll need to pick between speed or capacity. Major reports of faster ECC DIMMs seem to be only for lower capacity ones: 

https://www.tomshardware.com/pc-components/ddr5/overclockable-server-class-ecc-ram-options-expand-as-amd-threadripper-chips-arrive-gskill-introduces-zeta-r5-neo-ddr5-6400-rdimms

 

21 minutes ago, SG-O said:

All of this means that I will wait for a month or two until all components are released, more reviews are available and hopefully better ram is available.

That's your best option tbh. Anyhow, those benchmarks were enough to show that those TRs are beasts, however, AMX still has the upper hand for CPU inference if that's one major concern of yours.