Advice regarding a dual 4080 setup

[Legxcy]

I am looking for some advice about running a dual 4080 setup. I am currently running a 7900x with a 4080 and I am wanting to add another 4080 for Folding/BOINC (mainly PrimeGrid). The motherboard I have currently, only the 1st PCIe slot goes to the CPU so I am looking at upgrading, I have found a candidate, the MSI PRO X670-P, it has of course PCIe 4.0x16 slot going to the CPU and then it has another slot going back to the CPU which is PCIe 4.0x4.

 

My question is would the 4080 be able to use it's full potential at those speeds for Folding and BOINC or would I have to maybe go up to MSI MPG X670E CARBON which would allow me to run both the cards at PCIe 5.0x8 to the CPU.

 

Thanks in advance

[jaslion]

For folding and boinc you need basically no data throughput so just use the board you have now if it has a second slot for it call it a day

[Legxcy]

1 minute ago, jaslion said:

For folding and boinc you need basically no data throughput so just use the board you have now if it has a second slot for it call it a day

So what I have currently is a TUF Gaming B650 Plus, it only has 1 other x16 slot but that goes to the Chipset, do you think that'll still be fast enough. This is from the spec sheet. Thanks for your help

[jaslion]

1 minute ago, Legxcy said:

So what I have currently is a TUF Gaming B650 Plus, it only has 1 other x16 slot but that goes to the Chipset, do you think that'll still be fast enough. This is from the spec sheet. Thanks for your help

Yes the chipset pcie slot is barely "slower" it technically has more latency but you won't notice.

 

And for performance basically it's like mining you could have a pcie x1 link and it wouldn't care.

[Legxcy]

4 minutes ago, jaslion said:

Yes the chipset pcie slot is barely "slower" it technically has more latency but you won't notice.

 

And for performance basically it's like mining you could have a pcie x1 link and it wouldn't care.

Ok lovely. That saves me lots of money, just need a bigger case, a riser and the card now. Thank you

[Gorgon]

On 3/22/2023 at 8:08 PM, jaslion said:

Yes the chipset pcie slot is barely "slower" it technically has more latency but you won't notice.

 

And for performance basically it's like mining you could have a pcie x1 link and it wouldn't care.

Wrong Wrong Wrong.

 

PCIe x1 may work OK for PrimeGrid but it will definitely cripple the performance on a 4080 for Folding.

 

Try the PCIe3 x4 slot and if it doesn't appear to lower performance too much then you can save the cost on a motherboard upgrade.

 

Though with most cases you'll have to get creative and use a riser to get a GPU connected to the lower slot due to clearances to the Power Supply shroud with 3-slot width cards.

[jaslion]

2 hours ago, Gorgon said:

Wrong Wrong Wrong.

 

PCIe x1 may work OK for PrimeGrid but it will definitely cripple the performance on a 4080 for Folding.

 

Try the PCIe3 x4 slot and if it doesn't appear to lower performance too much then you can save the cost on a motherboard upgrade.

 

Though with most cases you'll have to get creative and use a riser to get a GPU connected to the lower slot due to clearances to the Power Supply shroud with 3-slot width cards.

Last thing I found was a 2080ti with pcie 3.0 x1 vs x4 vs x8 vs x16 and the difference was a hundred and something points between x1 and x16. x8 vs x16 no difference. x4 vs x16 no diff.

 

Now that ops board functions with a pcie 4.0 x4 bandwith there is no eay its being held back

[Gorgon]

null

1 hour ago, jaslion said:

Last thing I found was a 2080ti with pcie 3.0 x1 vs x4 vs x8 vs x16 and the difference was a hundred and something points between x1 and x16. x8 vs x16 no difference. x4 vs x16 no diff.

 

Now that ops board functions with a pcie 4.0 x4 bandwith there is no eay its being held back

The yield differences between PCIe3x1 and x4/x8/x16 will vary depending on the atom size of the Work Units and The number of cores on the GPU so a quick test will not show you any valid differences in performance. About the only way to measure this is to run at PCIe3x1 and x4 for a week or so each to get a good sampling of WUs and then to compare the yields of the same WUs between the two tests.

 

There are lots of threads over at the Folding at Home Support Form like this one and this one discussing this very issue.

 

In general Windows suffers more than Linux with limited PCIe lanes.

 

The last testing I did with core 21 on Windows and Linux comparing the yields moving from PCIe3 x8 to x4 showed:

Quote

The general observation is:

Under Windows a higher-end NVidia GPU on a PCIe3 x4 link will see a 20% decrease in Folding at Home Production in Points per Day (PPD) compared to the same card under Linux.

 

Things have improved under Windows with Core22 but the way Windows handles the PCI bus has not so there likely would still be more of an impact under Windows.

[Legxcy]

4 hours ago, Gorgon said:

null

The yield differences between PCIe3x1 and x4/x8/x16 will vary depending on the atom size of the Work Units and The number of cores on the GPU so a quick test will not show you any valid differences in performance. About the only way to measure this is to run at PCIe3x1 and x4 for a week or so each to get a good sampling of WUs and then to compare the yields of the same WUs between the two tests.

 

There are lots of threads over at the Folding at Home Support Form like this one and this one discussing this very issue.

 

In general Windows suffers more than Linux with limited PCIe lanes.

 

The last testing I did with core 21 on Windows and Linux comparing the yields moving from PCIe3 x8 to x4 showed:

Things have improved under Windows with Core22 but the way Windows handles the PCI bus has not so there likely would still be more of an impact under Windows.

My board runs it's second x16 slot at PCIe Gen 4x4 (3.0x8), although this slot does go to the chipset so would be limited with the link between CPU and PCH. have you done any testing running cards in 3.0x8? Thanks for your help

[Gorgon]

7 hours ago, Legxcy said:

My board runs it's second x16 slot at PCIe Gen 4x4 (3.0x8), although this slot does go to the chipset so would be limited with the link between CPU and PCH. have you done any testing running cards in 3.0x8? Thanks for your help

I don't have any 4000-series GPUs yet. They're just too expensive at current. I'm hoping the 4070 will launch at a reasonable price but I'm not holding my breath

 

Yes, I've tested running GPUs (2070 Super and 3080s) at PCIe3 x8 and they work fine. I use x570 motherboards and run 5 dual GPU rigs so most of my GPUs run at PCIe3 x8.

 

In your use case Zen4 uses 4 PCIe4 links to connect from the CPU to the b650 Chipset so effectively PCIe4 x4 or PCIe3 x8 so your upstream bandwidth between the Chipset and CPU might get limited by any other traffic to/from the CPU to devices connected to the Chipset (SSD, NVMe, USB controllers ...) but you will likely be fine.

 

Most, if not all, current GPUs can't use all the bandwidth in a PCIe4 x16 link let alone a PCIe5 x16 link and there are a couple of people here running 2x 4090 on x670/z690 motherboards so at PCIe5 x8/x8. Gaming (Loading Texture Maps etc.) will more fully utilize a PCIe connection than Folding will as Folding just has to send the current Frame's data to the GPU and retrieve the results and send checkpoints from the GPU to CPU.

 

Bottom line. You should be fine. You already have the hardware you need. You can install a monitoring too like Harlam's Folding Monitor (Hfm.net), which many of us use, and compare the performance between the 2 4090s and see for yourself if there is any difference in yield between the same WU running on either GPU and then use this to make an informed decision on upgrading your motherboard.

[Legxcy]

16 hours ago, Gorgon said:

I don't have any 4000-series GPUs yet. They're just too expensive at current. I'm hoping the 4070 will launch at a reasonable price but I'm not holding my breath

 

Yes, I've tested running GPUs (2070 Super and 3080s) at PCIe3 x8 and they work fine. I use x570 motherboards and run 5 dual GPU rigs so most of my GPUs run at PCIe3 x8.

 

In your use case Zen4 uses 4 PCIe4 links to connect from the CPU to the b650 Chipset so effectively PCIe4 x4 or PCIe3 x8 so your upstream bandwidth between the Chipset and CPU might get limited by any other traffic to/from the CPU to devices connected to the Chipset (SSD, NVMe, USB controllers ...) but you will likely be fine.

 

Most, if not all, current GPUs can't use all the bandwidth in a PCIe4 x16 link let alone a PCIe5 x16 link and there are a couple of people here running 2x 4090 on x670/z690 motherboards so at PCIe5 x8/x8. Gaming (Loading Texture Maps etc.) will more fully utilize a PCIe connection than Folding will as Folding just has to send the current Frame's data to the GPU and retrieve the results and send checkpoints from the GPU to CPU.

 

Bottom line. You should be fine. You already have the hardware you need. You can install a monitoring too like Harlam's Folding Monitor (Hfm.net), which many of us use, and compare the performance between the 2 4090s and see for yourself if there is any difference in yield between the same WU running on either GPU and then use this to make an informed decision on upgrading your motherboard.

Oh great thank you for the information, I really appreciate it. My backup incase the performance does get affected is I get the MSI MPG X670e Carbon which the first 2 slots run back to the CPU at 5.0x8. It's just if I can save money by not getting a mobo then that's what I'm going to do 

[Gorgon]

9 hours ago, Legxcy said:

Oh great thank you for the information, I really appreciate it. My backup incase the performance does get affected is I get the MSI MPG X670e Carbon which the first 2 slots run back to the CPU at 5.0x8. It's just if I can save money by not getting a mobo then that's what I'm going to do 

No problem. Do keep an eye on thermals though. If one of your cards is getting hot-boxed by the other may clock lower. A little testing locking graphics clocks (see the Profiling thread linked in my Sig) will save a lot of power (& heat ) with only a marginal decrease in yield which should allow you to run both cards at identical clocks and power-limits to get a better comparison.

 

Let us know what you discover.

[Legxcy]

On 3/25/2023 at 7:56 PM, Gorgon said:

No problem. Do keep an eye on thermals though. If one of your cards is getting hot-boxed by the other may clock lower. A little testing locking graphics clocks (see the Profiling thread linked in my Sig) will save a lot of power (& heat ) with only a marginal decrease in yield which should allow you to run both cards at identical clocks and power-limits to get a better comparison.

 

Let us know what you discover.

I've just ordered the other card, should be hopefully building it all on Friday. In the meantime I decided to test the PCIe lane situation by changing my lane speed in the bios to PCIe 2.0x16 which is comparable to 4.0x4 and I didn't seem to lose any performance. Atleast not on BOINC I didn't. I think that potential problem is not out the way but I'm still second guessing myself with the space inside the case. I'm gonna switch to the 7000d and then use a riser from the second PCIe slot to drop the card down another slot (atleast that's the plan). As the 7000d has 8 horizontal slots rather than 7 (5000d). I'm in uncharted territory here so I'm hoping it'll all work. The only alternative is to potentially watercool one of them (or both) but I've done that before either.

[Gorgon]

4 hours ago, Legxcy said:

I've just ordered the other card, should be hopefully building it all on Friday. In the meantime I decided to test the PCIe lane situation by changing my lane speed in the bios to PCIe 2.0x16 which is comparable to 4.0x4 and I didn't seem to lose any performance. Atleast not on BOINC I didn't. I think that potential problem is not out the way but I'm still second guessing myself with the space inside the case. I'm gonna switch to the 7000d and then use a riser from the second PCIe slot to drop the card down another slot (atleast that's the plan). As the 7000d has 8 horizontal slots rather than 7 (5000d). I'm in uncharted territory here so I'm hoping it'll all work. The only alternative is to potentially watercool one of them (or both) but I've done that before either.

I bough Hybrid GPUs and run 1 twin-axial GPU in the lower slot and the hybrid GPU in the upper slot. Worked great but 2-slot GPUs seem to be a thing of the past now with the exception of a 2-slot 4070 from one retailer.

[Legxcy]

On 3/29/2023 at 3:53 AM, Gorgon said:

I bough Hybrid GPUs and run 1 twin-axial GPU in the lower slot and the hybrid GPU in the upper slot. Worked great but 2-slot GPUs seem to be a thing of the past now with the exception of a 2-slot 4070 from one retailer.

So I got the Meshiy 2 XL case instead because it has 9 expansions slots rather than 8 (7000D). And I also got the card today but now I've hit another problem. The PCIe slot spacing on my mobo just is not big enough to have 1 card in the first one and the other in the second, and then also when I try to use a riser to drop it, I still can't get the card attached to the case due to things on the motherboard blocking the other end of the riser (the female end that the gpu goes into). Any ideas on what mobo I could get?

[Gorgon]

53 minutes ago, Legxcy said:

So I got the Meshiy 2 XL case instead because it has 9 expansions slots rather than 8 (7000D). And I also got the card today but now I've hit another problem. The PCIe slot spacing on my mobo just is not big enough to have 1 card in the first one and the other in the second, and then also when I try to use a riser to drop it, I still can't get the card attached to the case due to things on the motherboard blocking the other end of the riser (the female end that the gpu goes into). Any ideas on what mobo I could get?

You have the Asus TUF b650 Gaming Plus?

 

So 3-slot spacing? I'm assuming your existing 3080 is a 3+ slot card then?

 

I Don't even know if there's any b650 boards with 4-slot spacing. It was possible back in the day but no one is using the upper slot anymore as they need that space for the m.2 connected to the CPU to keep the traces short.

 

You want to use a riser on the lower slot?

 

While not ideal you could try installing the second card in the 3-slot vertical bay but you'll likely have to run with the side off until a more permanent solution is found.

 

Gigabyte b650 Aorus Elite AX - 4-slot spacing $219.99US

Gigabyte b650 Aorus Pro AX - 4-slot spacing $249.99US

Gigabyte b650 Aero G - 5-slot spacing $269.99US

Gigabyte b650e Aorus Master - 4-slot spacing ~$400US

ASRock Pro b650 Pro RS  - 5-slot Spacing! $219.99US

 

Or replace the Meshify 2 XL with a mining Frame:

 

[Legxcy]

6 minutes ago, Gorgon said:

You have the Asus TUF b650 Gaming Plus?

 

So 3-slot spacing? I'm assuming your existing 3080 is a 3+ slot card then?

 

I Don't even know if there's any b650 boards with 4-slot spacing. It was possible back in the day but no one is using the upper slot anymore as they need that space for the m.2 connected to the CPU to keep the traces short.

 

You want to use a riser on the lower slot?

 

While not ideal you could try installing the second card in the 3-slot vertical bay but you'll likely have to run with the side off until a more permanent solution is found.

 

Gigabyte b650e Aorus Master - 4-slot spacing

So the 2 4080s I have are the MSI Ventus 3x OC which is 63mm thick and then the other is the MSI Gaming X Trio 4080 which is 67mm thick. And yes that first m.2 slot is problem for every mobo as you lose that first expansion slot on the case. it's so pointless it being there.

[Gorgon]

3 minutes ago, Legxcy said:

So the 2 4080s I have are the MSI Ventus 3x OC which is 63mm thick and then the other is the MSI Gaming X Trio 4080 which is 67mm thick. And yes that first m.2 slot is problem for every mobo as you lose that first expansion slot on the case. it's so pointless it being there.

See update post above!

[Legxcy]

5 minutes ago, Gorgon said:

See update post above!

Yeah just looked at that motherboard and yes looks like increased spacing. Then there's also another problem with the IO at the bottom of the mobo like the frontpanel stuff interfering with the 2nd GPU. How would you get around that?  

[Gorgon]

9 minutes ago, Legxcy said:

Yeah just looked at that motherboard and yes looks like increased spacing. Then there's also another problem with the IO at the bottom of the mobo like the frontpanel stuff interfering with the 2nd GPU. How would you get around that?  

Right-angle adapters - let me find a link

 

FP Audio - something like this but with male on one end. These are called IDC (Insulation Displacement Connectors) Connectors. The Fans might be more of a problem.

 

Or - use a USB DAC and a Fan Hub off one of the other connectors.

 

The Front-Panel power, Reset and LED connections can usually be squished on befor the lower GPU is installed.

[Legxcy]

1 minute ago, Gorgon said:

Right-angle adapters - let me find a link

So I just look at the spec sheet for the motherboard, it seems there's 3 M.2s inbetween the 1st and second slot, that would put it at a gap of ~66mm. I would probably have to put the Ventus (63mm thick) one at top and then that would leave about 5mm gap between gpus.

[Legxcy]

10 minutes ago, Gorgon said:

Right-angle adapters - let me find a link

https://www.moddiy.com/products/Motherboard-Front-Panel-Header-Dupont-10-Pin-2.54mm-Angled-Connector.html?srsltid=AfAwrE6xuNTK-fqjbr74lTy_8LOcZVzRXEHJl8H1KDVv_M83Wjt7pTNQ_z8

 

Would this work?

[Gorgon]

8 minutes ago, Legxcy said:

So I just look at the spec sheet for the motherboard, it seems there's 3 M.2s inbetween the 1st and second slot, that would put it at a gap of ~66mm. I would probably have to put the Ventus (63mm thick) one at top and then that would leave about 5mm gap between gpus.

That GPU is just slightly wider than 3 slots:

So you should be fine with a 4-slot spacing. some of that 63mm is likely the backplate over-hanging the upper slot.

2 minutes ago, Legxcy said:

https://www.moddiy.com/products/Motherboard-Front-Panel-Header-Dupont-10-Pin-2.54mm-Angled-Connector.html?srsltid=AfAwrE6xuNTK-fqjbr74lTy_8LOcZVzRXEHJl8H1KDVv_M83Wjt7pTNQ_z8

 

Would this work?

Yes, but you'd have to un-solder the existing connector and solder that one on voiding your warranty. Buying a DAC or using the rear jack and a extension is probably easier

[Legxcy]

4 minutes ago, Gorgon said:

That GPU is just slightly wider than 3 slots:

So you should be fine with a 4-slot spacing. some of that 63mm is likely the backplate over-hanging the upper slot.

Yes, but you'd have to un-solder the existing connector and solder that one on voiding your warranty. Buying a DAC or using the rear jack and a extension is probably easier

There wouldn't be any solder, I think it's designed to just plug in but then I guess if that connector is the same length as the front panel connectors and then it also has the right angled pins then the clearance would still be an issue

[Gorgon]

4 minutes ago, Legxcy said:

There wouldn't be any solder, I think it's designed to just plug in but then I guess if that connector is the same length as the front panel connectors and then it also has the right angled pins then the clearance would still be an issue

No, that's not an adapter but a connector to be soldered onto a PCB

 

No, your correct but it might not get you enough space