pciebifurcation Bifurcation Conundrum: Seeking Mobo Advice for 3x GPU Build

[Dennettic]

Hi everyone,

 

I'm planning a PC build focused on machine learning, deep learning, local AI generation, and at least some AAA 1080p/1440p gaming. I've got most of the build together but need advice on choosing the right motherboard to ensure sufficient PCIe lanes for my triple GPU setup. Here's my build plan:

 

Planned Starter Build: https://pcpartpicker.com/list/bqjhHG

• CPU: AMD Ryzen 9 7950X3D 4.2 GHz 16-Core Processor
• GPU: MSI VENTUS 2X BLACK OC GeForce RTX 4060 Ti 16 GB Video Card
• Case: Corsair 6500D Airflow ATX Mid Tower Case
• PSU: be quiet! Straight Power 12 1000 W 80+ Platinum Certified Fully Modular ATX Power Supply
• RAM: TEAMGROUP T-Create Expert 64 GB (2 x 32 GB) DDR5-6000 CL34 Memory
• Cooling:
  • ARCTIC P14 PST 72.8 CFM 140 mm Fan
  • (2 x) ARCTIC P12 PST 56.3 CFM 120 mm Fans 5-Pack
  • ARCTIC Liquid Freezer III 56.3 CFM Liquid CPU Cooler
• Storage:
  • Silicon Power US75 4 TB M.2-2280 PCIe 4.0 X4 NVME Solid State Drive

 

Final Build Goal: https://pcpartpicker.com/list/37tsGP

• CPU: AMD Ryzen 9 7950X3D 4.2 GHz 16-Core Processor
• GPU: (3 x) MSI VENTUS 2X BLACK OC GeForce RTX 4060 Ti 16 GB Video Card
• Case: Corsair 6500D Airflow ATX Mid Tower Case
• PSU: be quiet! Straight Power 12 1000 W 80+ Platinum Certified Fully Modular ATX Power Supply
• RAM: (2 x) TEAMGROUP T-Create Expert 64 GB (2 x 32 GB) DDR5-6000 CL34 Memory
• Cooling:
  • ARCTIC P14 PST 72.8 CFM 140 mm Fan
  • (2 x) ARCTIC P12 PST 56.3 CFM 120 mm Fans 5-Pack
  • ARCTIC Liquid Freezer III 56.3 CFM Liquid CPU Cooler
• Storage:
  • (3 x) Silicon Power US75 4 TB M.2-2280 PCIe 4.0 X4 NVME Solid State Drive

 

My primary concern is that I'm looking for a motherboard that supports an x8/x8/x8 PCIe bifurcation setup or an alternative equivalent. Given that the Ryzen 9 7950X3D supports up to 24 direct PCIe lanes, it should be theoretically possible. However, I need a motherboard that can handle this configuration for all x16 slots and accommodate these three 2-slot GPUs without any physical constraints.

I know the 4060 has much hate as it only has a 128-bit bus rate. (and at that price point). However, given that running a multi GPU setup would require a reduction of available PCIe lanes per slot anyways, the bandwidth would be capped anyways. The problem is figuring out the sweet spot and if it exists with the available motherboards on the market. The sweet spot for this is presumably x8x8x8 and have everything else run through the chipset (Sure, the storage might get bottlenecked through the x4 dedicated lanes from the chipset to the CPU, but to me, that's an acceptable trade off). The following chart is just a reference to the bandwidth speeds in regarding the current predicament.

 

source: https://infohub.delltechnologies.com/en-us/p/next-generation-dell-poweredge-servers-designed-with-pcie-gen-5-to-deliver-future-ready-bandwidth/

 

I'm not sure if there is a board that can configure x8x8x8 with the AM5 slot.

According to this spreadsheet that I've seen thrown around; https://docs.google.com/spreadsheets/d/1NQHkDEcgDPm34Mns3C93K6SJoBnua-x9O-y_6hv8sPs/edit?pli=1&gid=0#gid=0,that doesn't seem to be the case.

But when reviewing the bifurcation compatibility from ASUS (see below), that spreadsheet seems to have incomplete data;

 

source: https://www.asus.com/support/faq/1037507/

 

Aside from that, I haven't been able to find anymore data, especially from the other manufacturers. I'm asking the community for help on this one.

I am aware that with these new x870(e) boards, Gen5 is to become standard, if that means that all the slots on the board will be Gen5 (somehow, I doubt this to be the case) then I can just run all the slots at x4 and everything would be peachy, but that seems fantastical. I'm certain that'll just mean there will always be at least one x16 and M.2 slot that's Gen5.

 

If anyone as any additional information, whether it's available data, personal experience, general advice or even if I'm just spouting backwards and misguided nonsense, please let me know. I'm open to hearing anything.

[Tetras]

The CPU can only do 8/8 at best with the primary graphics slots, 4 lanes are used for the chipset, 4 lanes are used for the M.2 and 4 for another M.2 or other slots. So, in order to have 8/8/8 from the CPU, you'd need to not use any lanes for the M.2, which would leave the board short in comparison to competing boards.

 

I think 8/8/4 is probably the best you'll get with X670E.

 

Note that bifurcation tables usually refer to the ability to split the lanes of the primary slot (e.g. for running multiple M.2 on an expansion card), not for splitting them out between different slots, since that is commonly hardwired and not configurable in the BIOS, whereas bifurcation often is.

[freeagent]

Yup, bifurcation will not help you with GPU's.

[Dennettic]

18 minutes ago, Tetras said:

The CPU can only do 8/8 at best with the primary graphics slots, 4 lanes are used for the chipset, 4 lanes are used for the M.2 and 4 for another M.2 or other slots. So, in order to have 8/8/8 from the CPU, you'd need to not use any lanes for the M.2, which would leave the board short in comparison to competing boards.

 

I think 8/8/4 is probably the best you'll get with X670E.

 

Note that bifurcation tables usually refer to the ability to split the lanes of the primary slot (e.g. for running multiple M.2 on an expansion card), not for splitting them out between different slots, since that is commonly hardwired and not configurable in the BIOS, whereas bifurcation often is.

I can't use the chipset for my storage? What if I switch to SATA? Is there no way to use the 24 "usable" lanes for just the x16 slots? Even the title from ASUS support page mentions the "Compatibility of PCIE bifurcation between Hyper M.2 series Cards and Add-On Graphic Cards" (https://www.asus.com/support/faq/1037507/)? Am I not understanding that correctly? Excuse me, this is my first modern pc build. it's been a long time and my first time doing it alone.

[freeagent]

Switching to sata will not help. No way to use 24 lanes on your x16 slots.

 

I have a Hyper M.2 as well.

[Imakuni]

Or, here's a better idea: you make a 2 GPU setup and throw away those lame 4060tis for more powerful GPUs. That will sidestep the lane problem while also making it easier to cool and consuming less energy.

[Dennettic]

5 minutes ago, freeagent said:

Switching to sata will not help. No way to use 24 lanes on your x16 slots.

 

I have a Hyper M.2 as well.

Damn. Not one manufacturer.... Hopefully x870(e) boards might have some potential with gen5. But I ain't gonna hold my breath. Probably better off having an AI design a MOBO for me to send over to PCBway lolol. Pipedream

[Dennettic]

4 minutes ago, Imakuni said:

Or, here's a better idea: you make a 2 GPU setup and throw away those lame 4060tis for more powerful GPUs. That will sidestep the lane problem while also making it easier to cool and consuming less energy.

The price per vram is the attractive selling point, especially for ML/DL. Gaming is secondary and Nvidia is overpriced. Would go team red, but it's trash for AI

[porina]

8 minutes ago, Imakuni said:

Or, here's a better idea: you make a 2 GPU setup and throw away those lame 4060tis for more powerful GPUs. That will sidestep the lane problem while also making it easier to cool and consuming less energy.

Following this up, a 4070 Ti Super is about 2x the compute performance of a 4060 Ti, for less than 2x the cost. You still get 16GB VRAM. 2x 4070 TiS isn't that much more than 3x 4060 Ti 16GB. If budget will be a problem there, then if the workload doesn't mind different speed GPUs, the 2nd one could still be a 4060 Ti 16GB.

[Dennettic]

Just now, porina said:

Following this up, a 4070 Ti Super is about 2x the compute performance of a 4060 Ti, for less than 2x the cost. You still get 16GB VRAM. 2x 4070 TiS isn't that much more than 3x 4060 Ti 16GB. If budget will be a problem there, then if the workload doesn't mind different speed GPUs, the 2nd one could still be a 4060 Ti 16GB.

The difference there is a total of 32gb of vram vs 48gb for nearly the same price.

[Tetras]

30 minutes ago, Dennettic said:

I can't use for my storage? What if I switch to SATA? Is there way to use the 24 "usable" lanes for just the x16 slots?

The lane usage is hard wired into the board so you can't change it manually, regardless of how many of those lanes you use for a M.2 slot.

 

The motherboard manufacturer would have to intentionally not use the CPU M.2 lanes to redistribute those lanes to a third graphics slot and the demand for that is just not there. I suppose there is a small chance they'd do that with Epyc 4004, but for a consumer board? I can't see the justification.

 

30 minutes ago, Dennettic said:

Even the title from ASUS support page mentions the " of PCIE bifurcation between Hyper M.2 series Cards and Graphic Cards" (https://www.asus.com/support/faq/1037507/)? Am I not understanding that correctly?

When ASUS talk about bifurcation they mean splitting the lanes of an existing hardwired slot, NOT redistributing lanes from the CPU.

 

If you use a Hyper M.2 expansion card, they put multiple M.2 slots on a single card and for all of those slots to work, you need to split the lanes on the PCI-E slot.

 

For example, you buy a Hyper M.2 card with 4x M.2 slots. You plug this card into your primary graphics slot. You then use the BIOS (bifurcation options) to split those 16 lanes into 4/4/4/4 so that each M.2 slot will operate with 4 lanes each.

[porina]

2 minutes ago, Dennettic said:

The difference there is a total of 32gb of vram vs 48gb for nearly the same price.

I'm not familiar with the area, but it is my understanding that you wouldn't be sharing the VRAM between different GPUs as a larger pool. Each one works on its own pool. Going off GPU will be choked quite badly by PCIe bandwidth. The 4060 Ti 16GB is 288GB/s on board vs 16GB/s over PCIe. Thus 16GB just has to be enough for whatever is on that GPU. As the 4070 TiS is about 2x the speed, you'll get through about 2x the work in a given time compared to a single 4060 Ti 16GB.

[Dennettic]

25 minutes ago, Tetras said:

The lane usage is hard wired into the board so you can't change it manually, regardless of how many of those lanes you use for a M.2 slot.

 

The motherboard manufacturer would have to intentionally not use the CPU M.2 lanes to redistribute those lanes to a third graphics slot and the demand for that is just not there. I suppose there is a small chance they'd do that with Epyc 4004, but for a consumer board? I can't see the justification.

 

When ASUS talk about bifurcation they mean splitting the lanes of an existing hardwired slot, NOT redistributing lanes from the CPU.

 

If you use a Hyper M.2 expansion card, they put multiple M.2 slots on a single card and for all of those slots to work, you need to split the lanes on the PCI-E slot.

 

For example, you buy a Hyper M.2 card with 4x M.2 slots. You plug this card into your primary graphics slot. You then use the BIOS (bifurcation options) to split those 16 lanes into 4/4/4/4 so that each M.2 slot will operate with 4 lanes each.

Yeah, I think I get it now. I appreciate the input. Just disappointed that not a single manufacturer has provided the physical lanes for x8x8x8 support on any board. Could see a lot of opportunity for more affordable decent ML/DL options without significant bottlenecking.

[Dennettic]

33 minutes ago, porina said:

I'm not familiar with the area, but it is my understanding that you wouldn't be sharing the VRAM between different GPUs as a larger pool. Each one works on its own pool. Going off GPU will be choked quite badly by PCIe bandwidth. The 4060 Ti 16GB is 288GB/s on board vs 16GB/s over PCIe. Thus 16GB just has to be enough for whatever is on that GPU. As the 4070 TiS is about 2x the speed, you'll get through about 2x the work in a given time compared to a single 4060 Ti 16GB.

You'd be surprised. Check out the success this guy had benchmarking 6 4060ti 16GB on a Threadripper board. 

If that peaks you're interest, you could also check out https://www.youtube.com/@HeatonResearch/featured. He's got a lot of info for ML/DL.

 

 

For the 4060ti 16gb, it's no doubt terrible for gamming, but AI enthusiasts are generally very welcoming to the card. VRAM and CUDA is key for running larger models more efficiently. At least according to what I have been reading a watching. Would like to get some hands on data, TBH.

 

EDIT: I should also mention that speed isn't everything. You need more VRAM for running larger models. You could cap yourself at X VRAM for running a smaller model slightly faster (like by seconds) vs running at Y or much larger models for comparable speeds. Sure, you could have a dumber response faster, or you could have a smarter response at a "slower" speed (again, by seconds).

 

There is a trade off though, the key is to not go so slow that the response time isn't fluent, say in like a modded video game (Check out Skyrim's Herika), but want those responses to make sense and even be impressive.

Edited June 15, 2024 by Dennettic

[porina]

13 minutes ago, Dennettic said:

You'd be surprised. Check out the success this guy had benchmarking 6 4060ti 16GB on a Threadripper board. 

This isn't my interest area so I'll skip it. It may be my understanding is wrong, but see if you can find someone who has compared 2x 4060 Ti 16GB vs 1x 4070 Ti Super. It might not exist. In theory both should perform the same. 

 

13 minutes ago, Dennettic said:

For the 4060ti 16gb, it's no doubt terrible for gamming, but AI enthusiasts are generally very welcoming to the card. VRAM and CUDA is key for running larger models more efficiently. At least according to what I have been reading a watching. Would like to get some hands on data, TBH 

Again, VRAM matters to the card it is fitted on. It shouldn't matter how much total you have separately in a system until you get to high end products using nvlink which is its own high bandwidth connection separate from PCIe. I picked the 4070 Ti Super specifically because it has the 16GB to match a 4060 Ti 16GB, but it should do work in half the time.

[Ralf]

50 minutes ago, Dennettic said:

Damn. Not one manufacturer.... Hopefully x870(e) boards might have some potential with gen5. But I ain't gonna hold my breath. Probably better off having an AI design a MOBO for me to send over to PCBway lolol. Pipedream

X870E is basically the same ad X670E, only difference is that X870E uses 4 cpu lanes for USB4.

[Tetras]

35 minutes ago, Dennettic said:

Yeah, I think I get it now. I appreciate the input. Just disappointed that not manufacturer has provided the physical lanes for x8x8x8 support on any board. Could see a lot of for more affordable decent ML/DL options without significant bottlenecking.

Enthusiasts have complained about there not being an affordable HEDT platform with lots of lanes since X99, but for now, you still have to get threadripper or xeon if you want lots of PCI-E lanes, e.g. W790.

[Dennettic]

13 minutes ago, porina said:

This isn't my interest area so I'll skip it. It may be my understanding is wrong, but see if you can find someone who has compared 2x 4060 Ti 16GB vs 1x 4070 Ti Super. It might not exist. In theory both should perform the same. 

 

Again, VRAM matters to the card it is fitted on. It shouldn't matter how much total you have separately in a system until you get to high end products using nvlink which is its own high bandwidth connection separate from PCIe. I picked the 4070 Ti Super specifically because it has the 16GB to match a 4060 Ti 16GB, but it should do work in half the time.

Speed isn't everything. For ML/DL and AI generation. You want to be able to run larger models and datasets. You may be able to run a faster speed on the 4070 ti but it's speed difference is measured in seconds compared to the 4060ti 16gb. However, having more vram in total will allow you to run larger models. If I had only a single 4070ti super, then I would only be able to run models sizes that fit into 16gb. If I have two 4060ti then I can run models that fit within 32gb. The speed differences is nominal in my opinion and use case. (I'd say otherwise if I was running the largest datasets like the big companies do, you want every bit of performance at that scale)

[Dennettic]

13 minutes ago, Ralf said:

X870E is basically the same ad X670E, only difference is that X870E uses 4 cpu lanes for USB4.

That's a shame, but not surprising. I was only bringing it up because they're vague claim that PCIE gen5 will be "Standard" on all mobos. Wishing for more but expecting less seem to be a safe bet for these manuf.s

[manikyath]

do your GPU's all need propper bandwidth, or can you get away with using let's say 2 lanes for the cards that are only used for machine learning? 

 

if you greatly prefer propper bandwith, look into scoring something epyc 7002/7003 series, with a motherboard that exposes most of the lanes into PCIe slots, at that point it's just a matter of plonking the cards in and calling it a day.

downside of epyc is that you really also want audio and USB add-in cards, because by design the platform's options are limited by design.

 

alternatively you could look into threadripper, but pricing really just sucks on that side.

[Dennettic]

9 minutes ago, manikyath said:

do your GPU's all need propper bandwidth, or can you get away with using let's say 2 lanes for the cards that are only used for machine learning? 

 

if you greatly prefer propper bandwith, look into scoring something epyc 7002/7003 series, with a motherboard that exposes most of the lanes into PCIe slots, at that point it's just a matter of plonking the cards in and calling it a day.

downside of epyc is that you really also want audio and USB add-in cards, because by design the platform's options are limited by design.

 

alternatively you could look into threadripper, but pricing really just sucks on that side.

It's the pricing for me. Especially having to get older hardware at the price point. Might be better getting a cheaper board until gen4 becomes obsolete, and then start upgrading then. Hopefully that's sooner rather than later, but I don't got much hope for that.

[Dennettic]

21 minutes ago, Tetras said:

Enthusiasts have complained about there not being an affordable HEDT platform with lots of lanes since X99, but for now, you still have to get threadripper or xeon if you want lots of PCI-E lanes, e.g. W790.

[manikyath]

Just now, Dennettic said:

It's the pricing for me. Especially having to get older hardware at the price point. Might be better getting a cheaper board until gen4 becomes obsolete, and then start upgrading then. Hopefully that's sooner rather than later, but I don't got much hope for that.

well, that's why i ask; does the bandwidth matter?

 if bandwidth doesnt matter, you do some BS wizardry with a board that has a bunch of x2 or x4 options, if bandwidth matters, going with epyc really is the way to go.

 

on the note of BS wizardry.. i just ran into this board that has 3 slots, topmost can hold the GPU you use for gaming, other two slots are essentially limited to PCIe 4.0 x2, so if you're fine with x2, you just go that route:

https://www.msi.com/Motherboard/MAG-X670E-TOMAHAWK-WIFI

 

4 minutes ago, Dennettic said:

-blackjack meme-

 

epyc, it's so packed with features it almost looks like a joke. yes.. asrock managed to squeeze an SP3 socket, 8 DIMM slots, and all 7 PCIe 16x slots onto a standard ATX board.

[porina]

46 minutes ago, Dennettic said:

Speed isn't everything. For ML/DL and AI generation. You want to be able to run larger models and datasets. 

Again this is not an area I'm very familiar with, but I still doubt you can pool the VRAM of separate consumer GPUs in that way. The high end workstation/professional GPUs that can pool VRAM have nvlink between GPUs providing substantially more bandwidth than PCIe could alone. As such it is my understanding you'll just be running multiple instances of up to 16GB.

 

Not asking you to believe me but it is worth checking out in case you waste your money on something that doesn't work how you think it does.

[mariushm]

No, I don't think the memory can't be pooled together.  I was gonna say in the reply that 4060 cards are a bad idea not due to the 128 bit bus, but due to the low amount of vram.

 

MSI and others are working with Phison these days to make a sort of nvme to ram solution, basically they're making a bunch of nvme drives work as ram, so the video card would be able to transparently swap big blocks of data to-from a bunch of nvme drives

 

As others have told you bifurcation is about the manufacturer making it possible to split the lanes in a physical slot into 2 or more groups of pci-e lanes... basically, split a x16 slot into 4 groups of 4 lanes - see cards like Hyper from Asus that create 4 m.2 each with 4 lanes by bifurcating the x16 slot into 4 x4 slots.

 

There were some motherboards that did this split on purpose, I remember a Gigabyte one on socket AM4 that had only two pci-e x16 slots but both had only 8 pci-e lanes (so they split the x16 into 2 x8) and they also split the second x8 into 2 m.2 x4  ... so the video card would always run at x8 no matter what, and the second x8 slot would be disabled if you used any of the 2 m.2 slots.  The upside is you had 3 m.2 slots connected directly to cpu and with 4 lanes each, downside is video card has only 8 lanes.

 

Epyc boards and cpus are a good compromise, they'll offer loads of ram slots (but you have to get Registered DDR4 which won't be compatible with desktop systems) and there's cheap (sort of)  Epyc processors similar in performance to 7950x3d  though it's debatable that you need such high end cpu, because most AI stuff will be done on the video cards.

 

There's a seller on eBay that sells a lot of cpus and mobos, heard of him from servethehome forums, has a decent reputation there

Here's the page : https://www.ebay.com/str/tugm4470/Computer-Components-Parts/_i.html?_sacat=175673

 

Same seller will also have coolers and ram and may give you an offer if you buy all from him as a package