PCie Lanes Why not enough slots?

[TechXero]

Hey there...
 

A friend of mine who uses a lot of add-in cards one day asked me a question I couldn't answer even after Googling for a while. I was wondering if anyone here can shed some light...
 

What AMD CPU provides most number of PCie Lanes and why isn't there an AMD Chipset Mobo with more than 4 Full length PCie Slots?

 

He currently is on Intel with a 5960x / WS mobo with 6 PCie Slots all of which are currently being used...

 

Thanks in advance.

[eeeee1]

only the MSI Godlike has 4 16x slots and that is the most you can get.

 

and the price is 1049.99 CAD

[TechXero]

2 minutes ago, eeeee1 said:

only the MSI Godlike has 4 16x slots and that is the most you can get.

 

and the price is 1049.99 CAD

So he might as well stick with Intel. What a shame... But can anyone shed some light as to why? 

 

And he doesn't care if it's x8 or x16 just not x4 or below

[Alex Atkin UK]

31 minutes ago, TechXero said:

So he might as well stick with Intel. What a shame... But can anyone shed some light as to why? 

 

And he doesn't care if it's x8 or x16 just not x4 or below

Because there are far less lanes than the total number of slots requires, so they have to be multiplexed.  Generally all the PCIe x4 slots are combined off a single connection to the chipset which on Intel is comparable to a single PCIe x4 connection, on AMD it may be slighter more bandwidth than that, I'm not familiar with how AMD works.

Generally the x16 slots aren't, they are direct CPU and just downgrade as you add more cards.  So one card is x16, two cards are x8 x8, three cards x8 x4 x4. (the specific allocation can vary between motherboards)

The more lanes you need to multiplex, the more expensive and complicated the motherboard will be, plus a very real risk of cards starting to bottleneck each other.

 

As most people only have a GPU, the manufacturer avoid adding extra multiplexing to their boards that will never be used.

[igormp]

Usually only HEDT platforms offer enough PCIe lanes to justify having multiple full size PCIe slots on the mobo, such as Xeons and Threadrippers. 

 

However, if all you need are the slots themselves at lower speeds, some b550 and x570 mobos allow for easy PCIe bifurcation, meaning that you can slap a splitter into a x16 slot and have 4 x4 cards coming out of a single slot, along with the other existing x4 slot from the CPU and the ones from the chipset.

[Alex Atkin UK]

Also there is only so much physical space, so most motherboard manufacturers trade x16 slots for M.2 slots and other components they need to squeeze in there.

HEDT platforms however have less built-in to the motherboard, understanding that you will instead use PCIe cards to replicate that functionality with higher-grade hardware.

[TechXero]

5 minutes ago, igormp said:

Usually only HEDT platforms offer enough PCIe lanes to justify having multiple full size PCIe slots on the mobo, such as Xeons and Threadrippers. 

 

However, if all you need are the slots themselves at lower speeds, some b550 and x570 mobos allow for easy PCIe bifurcation, meaning that you can slap a splitter into a x16 slot and have 4 x4 cards coming out of a single slot, along with the other existing x4 slot from the CPU and the ones from the chipset.

Well clearly there are people like my friend here who use more than 4 slots. He has 2x1080Ti 1x950 1x4Port 1G Nic 1xHBA and a 4 port 10G Nic card... He's looking for an AMD board where he can use all those and they are all full length 

[Alex Atkin UK]

2 minutes ago, TechXero said:

Well clearly there are people like my friend here who use more than 4 slots. He has 2x1080Ti 1x950 1x4Port Nic 1xHBA and a 8 USB 3.0 hub card... He's looking for an AMD board where he can use all those and they are all full length 

He would probably have to go threadripper to get those slots.

[TechXero]

Just now, Alex Atkin UK said:

He would probably have to go threadripper to get those slots.

Ahh... We looked... None to be found... All offer 4 ports max

[Alex Atkin UK]

6 minutes ago, TechXero said:

Ahh... We looked... None to be found... All offer 4 ports max

Yes it seems they too sacrifice slots for M.2 sockets.  It seems 5 slots is the best you can get. https://www.gigabyte.com/uk/Motherboard/X399-DESIGNARE-EX-rev-10#kf

I'd imagine waiting for Zen 3 Threadripper is a good idea anyway.

[TechXero]

1 minute ago, Alex Atkin UK said:

Yes it seems they too sacrifice slots for M.2 sockets.  A quick Google finds a 5 slot variant. https://www.gigabyte.com/uk/Motherboard/X399-DESIGNARE-EX-rev-10#kf

I'd imagine waiting for Zen 3 Threadripper is a good idea anyway.

If there was a brand we could kill it would be Gigabyte coz we had nothing but bad experience with their products... That aside he wants AMD not Intel... So yeah wait n see I guess...

[Alex Atkin UK]

19 minutes ago, TechXero said:

If there was a brand we could kill it would be Gigabyte coz we had nothing but bad experience with their products... That aside he wants AMD not Intel... So yeah wait n see I guess...

Not had any problems personally, I don't really have a preferred brand although seem to mostly end up with ASUS.  Apparently they are awful if you need warranty support, but I never have.

[mariushm]

There's 24 pci-e lanes coming from cpu.

16 of those go to video card slot and motherboard manufacturer can choose to split it in 2 pci-e x8 slots.

4 go to the m.2 connector

4 go to chipset

 

The chipsets creates 6-8 pci-e lanes. Usually 4 go to the bottom pci-e x16 slot, and the others go to pci-e x1 slots and additional m.2 connectors.

 

The  cpu can't have more pci-e lanes due to socket am4 limitation (number of pins).

With pci-e 4.0, there's also a limitation regarding maximum length from the cpu pins to the gpu chip on a card, or another chip on whatever card you insert in slot.  You have only around 30-40 cm, so if you have a slot too low on the board, the motherboard manufacturer needs to add amplifier/repeater chips which cost money.

 

Motherboard manufacturers could also use  pci-e switch chips, which take in 4 / 8 / 16 pci-e lanes and create 16/24/32 pci-e lanes, which can be arranged in various configuration. However, there's few manufacturers of such chips and they're quite expensive, almost hitting the price of Threadripper boards.

 

Threadripper processors come with 64 pci-e lanes, so there's 60 pci-e lanes available for pci-e slots and m.2 connectors. A lot of motherboards will have 4 or 5 pci-e x16 slots and 2-3 m.2 connectors.

 

 

[TechXero]

7 hours ago, mariushm said:

Threadripper processors come with 64 pci-e lanes, so there's 60 pci-e lanes available for pci-e slots and m.2 connectors. A lot of motherboards will have 4 or 5 pci-e x16 slots and 2-3 m.2 connectors.

 

 

Exactly... But no motherboards give you 5 PCie Slots we have looked high and low... Why?

[mariushm]

2 hours ago, TechXero said:

Exactly... But no motherboards give you 5 PCie Slots we have looked high and low... Why?

Because physically they would take up too much room on the motherboard, making it difficult to add m.2 connectors and/or other things on the board.

 

 

My favorite IT store lists 3 motherboards that have 5 pci-e x16 slots :

 

x16/x16/x8/x4/x4 :

 

1. GIGABYTE X299 UD4 PRO X299 UD4 Pro (rev. 1.0) Key Features | Motherboard - GIGABYTE Global

2. GIGABYTE X299 DESIGNARE EX  X299 DESIGNARE EX (rev. 1.0) Key Features | Motherboard - GIGABYTE Global

 

x16 / x16 / x8 / x8 / x4:

 

1. GIGABYTE X399 AORUS PRO X399 AORUS PRO (rev. 1.0) Key Features | Motherboard - GIGABYTE Global

 

With 4 pci-e x16 slot, the list expands to around 15-20 motherboards, and i'm sure my local store has a small selection of what's out here -  and you could use a pci-e riser cable if you absolutely have to install a 5th card, of course resulting in lower bandwidth to that slot.

 

ASRock X399 Taichi
ASRock Z270 SuperCarrier
ASUS PRIME X399-A
ASUS ROG STRIX X399-E GAMING
ASUS ROG ZENITH X399 EXTREME
ASUS ROG ZENITH X399 EXTREME ALPHA
GIGABYTE X299 AORUS MASTER
GIGABYTE X299X AORUS MASTER
GIGABYTE X299X DESIGNARE 10G
GIGABYTE X399 AORUS XTREME
GIGABYTE X399 Designare EX
MSI X299 GAMING PRO CARBON
MSI X299 PRO
MSI X299 PRO 10G
MSI X299 RAIDER
MSI X299 SLI PLUS
MSI X299 TOMAHAWK AC
MSI X299 XPOWER GAMING AC
MSI X399 SLI PLUS

 

Note that none of the above are worth buying, because both chipsets (x299 and x399) are for sockets that are "outdated", there's newer threadripper which uses the other socket and also on intel... socket 2066 is done.

 

 

 

 

 

 

 

 

[TechXero]

2 minutes ago, mariushm said:

Because physically they would take up too much room on the motherboard, making it difficult to add m.2 connectors and/or other things on the board.

 

 

My favorite IT store lists 3 motherboards that have 5 pci-e x16 slots :

 

x16/x16/x8/x4/x4 :

 

1. GIGABYTE X299 UD4 PRO X299 UD4 Pro (rev. 1.0) Key Features | Motherboard - GIGABYTE Global

2. GIGABYTE X299 DESIGNARE EX  X299 DESIGNARE EX (rev. 1.0) Key Features | Motherboard - GIGABYTE Global

 

x16 / x16 / x8 / x8 / x4:

 

1. GIGABYTE X399 AORUS PRO X399 AORUS PRO (rev. 1.0) Key Features | Motherboard - GIGABYTE Global

 

With 4 pci-e x16 slot, the list expands to around 15-20 motherboards, and i'm sure my local store has a small selection of what's out here -  and you could use a pci-e riser cable if you absolutely have to install a 5th card, of course resulting in lower bandwidth to that slot.

 

ASRock X399 Taichi
ASRock Z270 SuperCarrier
ASUS PRIME X399-A
ASUS ROG STRIX X399-E GAMING
ASUS ROG ZENITH X399 EXTREME
ASUS ROG ZENITH X399 EXTREME ALPHA
GIGABYTE X299 AORUS MASTER
GIGABYTE X299X AORUS MASTER
GIGABYTE X299X DESIGNARE 10G
GIGABYTE X399 AORUS XTREME
GIGABYTE X399 Designare EX
MSI X299 GAMING PRO CARBON
MSI X299 PRO
MSI X299 PRO 10G
MSI X299 RAIDER
MSI X299 SLI PLUS
MSI X299 TOMAHAWK AC
MSI X299 XPOWER GAMING AC
MSI X399 SLI PLUS

 

Note that none of the above are worth buying, because both chipsets (x299 and x399) are for sockets that are "outdated", there's newer threadripper which uses the other socket and also on intel... socket 2066 is done.

 

 

 

 

 

 

 

 

Wgen the guy wants AMD he is unlucky... So bottom line stick with Intel

[mariushm]

5 minutes ago, TechXero said:

Wgen the guy wants AMD he is unlucky... So bottom line stick with Intel

Well if he wants AMD, like i said there's x399 ... for the older Threadripper.

Yes, there's more x299 boards for socket 2066, but they're not all x16 slots, they're mostly x16 and the rest x8 or x4, because there's not enough pci-e lanes on x299 boards.

 

On the latest threadripper boards you have boards like GIGABYTE TRX40 AORUS PRO WIFI  or GIGABYTE TRX40 DESIGNARE  with 4 pci-e x16 ( 2x16 and 2x8) and one pci-e x1 slot .... also they're all pci-e 4.0,  so a pci-e 4.0 x8 is prety much equivalent to pci-e 3.0 x16

 

One more thing I forgot about why not 5 or more pci-e x16 slots - because most people that would buy such expensive boards would buy powerful video cards that take up 2-3 slots, so the room below the first pci-e slot will usually be used for other things like m.2 connector or extra controllers.

 

You can check out that TRX40 designare to see what i mean, those 4 pci-e slots are perfectly spaced, so you could install 4 double slot video cards.

 

 

 

[-iSynthesis]

If money is no concern he could go Epyc and go with that ASRock Rack Mobo with 7 PCIe x16 slots (ROMED8-2T)

[TechXero]

Thanks for all the info guys. But you all got the assumption that what he is asking for is 4 or more x16 slots he doesn't care if they are x8... Just not x4 coz all his add-in cards are long like 4 port Nic card and GPUs. For GPUs 1x16 to main 2x4k Monitors and 1 dedicated GPU for 3rd always on screen a 960 it's just so he can check what's happening on system. So doesn't matter 8f it is running in x8... It's ports he cares about coz if he upgrades now with boards lacking slots he won't be able to use all his cards. So he is stuck...

[mariushm]

1 hour ago, TechXero said:

Thanks for all the info guys. But you all got the assumption that what he is asking for is 4 or more x16 slots he doesn't care if they are x8... Just not x4 coz all his add-in cards are long like 4 port Nic card and GPUs. For GPUs 1x16 to main 2x4k Monitors and 1 dedicated GPU for 3rd always on screen a 960 it's just so he can check what's happening on system. So doesn't matter 8f it is running in x8... It's ports he cares about coz if he upgrades now with boards lacking slots he won't be able to use all his cards. So he is stuck...

Unless those are 10gbps ports, the card should run even in a x1 slot, though x2 or x4 would be preferred.

 

A 1 gbps port does 125 MB/s so 4 ports would do a maximum of 500 MB/s

pci-e 2.0 x1  has a bandwidth of 500 MB/s,  pci-e 3.0 x1 does up to 970 MB/s - in real world we're talking around 480 MB/s or 950 MB/s real bandwidth, after pci-e overhead and all that crap.

 

So if it's an older chip on that 4 port gigabit card that does only pci-e 2.0, then at least pci-e x2 should be used, but even x1 would work because it's really unlikely all ports will be at 100% at the same time.

 

4 10g ports would need 5 GB/s which is around equivalent to 6 pci-e lanes ( 6 x 970 = 5820 MB/s) but you could live with 4 pci-e 3.0 (~3.8 GB/s) if not all 10g ports will be used at same time.

[TechXero]

3 minutes ago, mariushm said:

Unless those are 10gbps ports, the card should run even in a x1 slot, though x2 or x4 would be preferred.

 

A 1 gbps port does 125 MB/s so 4 ports would do a maximum of 500 MB/s

pci-e 2.0 x1  has a bandwidth of 500 MB/s,  pci-e 3.0 x1 does up to 970 MB/s - in real world we're talking around 480 MB/s or 950 MB/s real bandwidth, after pci-e overhead and all that crap.

 

So if it's an older chip on that 4 port gigabit card that does only pci-e 2.0, then at least pci-e x2 should be used, but even x1 would work because it's really unlikely all ports will be at 100% at the same time.

 

4 10g ports would need 5 GB/s which is around equivalent to 6 pci-e lanes ( 6 x 970 = 5820 MB/s) but you could live with 4 pci-e 3.0 (~3.8 GB/s) if not all 10g ports will be used at same time.

My friend :

Other cards, including HBA use 8, And 4 port gigabit uses 4. Lanes that is.. Like I said he got tons of cards.. None of his cards fit an x4 slot..

[mariushm]

pci-e is designed from the start to be modular.  Pretty much all devices will work with fewer lanes.

If the slot doesn't have a physical wall and there's no components obstructing the path (what's after the slot) you can insert a pci-e x8 or pci-e x16 card into a pci-e x1 or pci-e x4 slot leaving a part of the edge connector floating in the air, unconnected to anything.

Both video card and slot will properly detect the lower number of pci-e lanes and the card should work just fine.

Some motherboards will actually be made with no wall at the end of the pci-e slot, to allow bigger cards.

 

For this reason, a lot of low profile cards can be used with a tiny adapter or small pci-e riser cable to change a smaller slot to a bigger slot.

 

For example this adapter converts a pci-e x1 slot to pci-e x16, so you can plug low profile cards into it :

Amazon.com: StarTech.com PCI Express X1 to X16 Low Profile Slot Extension Adapter - PCIe x1 to x16 Adapter (PEX1TO162): Electronics

 

Here's the riser cable version, which allows you to add slots under the motherboard, for example if you have a mATX board in an ATX case : Amazon.com: ADT-LINK PCIE 4X Riser Cable Dual 90 Degree Right Angle PCIe 3.0 x1 to x4 Extension Cable 8Gbps PCI Express 1x Riser Card Ribbon Extender (10CM, R12SL-FL): Computers & Accessories

 

Notice how the pci-e x4 slot doesn't have a wall at the end, allowing you to install a pci-e x8 or x16 card in the slot, but it will still run at pci-e x1.

 

Here's a pci-e x4 to pci-e x4 (without wall, allowing for x8 and x16 cards) small riser cable:

Amazon.com: ADT-Link PCIe 3.0 x4 Extension Cable 32G/BPS PCI Express 4X Graphic SSD RAID Extender Conversion Riser Card Vertical 90 R22SL(5CM): Computers & Accessories

 

Anyway... the HBA adapter card will happily work in a pci-e x4 slot, unless you're using only SSDs on it, you're unlikely to need 8 pci-e lanes worth of bandwidth.

Same for secondary video card, if it's just for additional displays and maybe cuda encoding, 4 pci-e lanes are plenty.