PCI4 lane count with PCI3 devices

[Overtaxed]

I'm running short of PCI lanes in my X370 motherboard (video card, NVME drive, 10G card, add on SATA adapter) and I'm wondering what happens when you overcommit lanes?  If I have 24 lanes available, but plug in 32 lanes worth, does it just drop back one of the X16 slots automatically?  

 

Also, I'm kind of curious, my CPU is a 3950 which has PCI4.  My motherboard is PCI3, but if I move to a PCI4 motherboard is the number of lanes doubled if everything plugged in is PCI3?  Does a X16 PCI3 video card only take X8 PCI4 lanes?  

 

Sorry, I realize these questions have probably been answered before, but I couldn't find anything that directly spoke to this type of situation.

[WhitetailAni]

Just now, Overtaxed said:

Also, I'm kind of curious, my CPU is a 3950 which has PCI4.  My motherboard is PCI3, but if I move to a PCI4 motherboard is the number of lanes doubled if everything plugged in is PCI3?  Does a X16 PCI3 video card only take X8 PCI4 lanes?  

Unfortunately, it doesn't work like that.

An x16 PCIe 3.0 card will not run at 4.0 x8 since it doesn't support 4.0 x8. Think of it as if each lane is a highway lane. Adding more makes it so more cars can go through, and going to another generation makes the speed limit higher.

A device at the other end will have a speed and highway size limit that cannot be increased beyond a certain point - a 3.0 x16 GPU in this case. If we set the speed limit to PCIe 4.0, the "cars" will go at 4.0 speed, but then drop down to 3.0 speed once they enter the GPU portion of the "highway". If we reduce the lane size to x8, there will be half the highway space and thus half the bandwidth.

I hope this makes sense!

[Overtaxed]

Sadly, that's kind of what I thought.  So, put another way, if I could find a PCI4 x8 card with the same specs as my PCI3 x16 card, I'd basically free up 8 lanes for another use?  If there's only 24 lanes, would seem the best path (using your analogy) is to make sure every car is driving as fast as possible and as "narrow" (least number of lanes) as possible while at that high speed?  You know the strange thing though, there are at least some cards that are available in PCI 2 and 3 (NIC cards, for example) where they use X4 connections in both standards.  That makes no sense, a PCI 4 video card should really be X8 if it has the same bandwidth needs as a PCI 3 X16 card, right?  Would seem like that would be a really good idea to free up lanes.  

 

Any idea on the other part of the question?  What happens if, for example, I plug in a X16 video card and then drop in a X16 quad NVME (bifurcation) card in the 2nd slot?  Does it "share" the lanes between them, or does something drop back to X8?  I know when you put in more video cards, it drops back, just not sure what happens if you put in a quad NVME, a X16 video card and lets say a X4 10G NIC, something is going to be starved for lanes, does it just figure it out or do things stop working (like you lose access to some of the NVME slots)?

[bowrilla]

25 minutes ago, Overtaxed said:

I'm running short of PCI lanes in my X370 motherboard (video card, NVME drive, 10G card, add on SATA adapter) and I'm wondering what happens when you overcommit lanes?  If I have 24 lanes available, but plug in 32 lanes worth, does it just drop back one of the X16 slots automatically?  

No, not working like that (at least not automatically). Ryzen has 24 CPU lanes: 16x for the GPU, 4x for the first/primary NVMe M.2 slot and 4x for the the chipset uplink, which handles everything like additional USB ports, SATA ports, ethernet ports and additional PCIe slots (potentially up to 16 additional lanes here). The catch is, that those PCIe lanes WILL need to go through the uplink/connection of CPU and chipset (just like EVERYTHIGN else that's connected through the chipset). So all data that needs to go to the CPU will share those 4x lanes between chipset and CPU. No way around it.

 

Some mainboards offer PCIe bifurcation or some sort of onboard switching. With that you can split the 16x GPU lanes into 8x + 8x for exmaple or 8x + 4x + 4x (or 4 time 4x). Obviously this will potentially impact your GPU performance.

 

25 minutes ago, Overtaxed said:

Also, I'm kind of curious, my CPU is a 3950 which has PCI4.  My motherboard is PCI3, but if I move to a PCI4 motherboard is the number of lanes doubled if everything plugged in is PCI3?  Does a X16 PCI3 video card only take X8 PCI4 lanes? 

Nope, unfortunately not. Electrically PCIe gen3 and gen4 are identical. It's just that PCIe gen4 can use more bandwith. Obviously with bifurcation of PCIe gen4 16x into 8x + 8x  you will not suffer such a big if any impact in GPU performance compared to PCIe gen3. BUT your GPU will need to be PCIe gen4 compatible. After all you will end up with only 8 lanes.

 

It's sort of the same with ehternet speeds or internet speeds: it goes through the same cable but different standards offer different speeds (for various reasons).

[WhitetailAni]

Make sure to quote people so they get notifications.

5 minutes ago, Overtaxed said:

So, put another way, if I could find a PCI4 x8 card with the same specs as my PCI3 x16 card, I'd basically free up 8 lanes for another use?

Yes, but unless your motherboard supports PCIe 4.0, your card will run a PCIe 3.0 x8, not 4.0 x8 (I think).

5 minutes ago, Overtaxed said:

If there's only 24 lanes, would seem the best path (using your analogy) is to make sure every car is driving as fast as possible and as "narrow" (least number of lanes) as possible while at that high speed?

Don't forget about chipsets, too. Some PCIe lanes are routed through your chipset, which has a link to your CPU. That's why motherboards can have more than an x16 slot and 2 M.2 x4 slots.

6 minutes ago, Overtaxed said:

That makes no sense, a PCI 4 video card should really be X8 if it has the same bandwidth needs as a PCI 3 X16 card, right?  Would seem like that would be a really good idea to free up lanes.

I'm not an expert on how PCIe lane allocation works, but I believe PCIe can do that automagically if something else needs more lanes.

7 minutes ago, Overtaxed said:

What happens if, for example, I plug in a X16 video card and then drop in a X16 quad NVME (bifurcation) card in the 2nd slot?

Like the Hyper M.2?
Only two drives would work, since only 8 lanes would be connected. The others would not be connected.

 

[Overtaxed]

Makes sense, thanks for the details.  Let me ask something though, how on earth does someone use a card like this:

https://www.amazon.com/ASUS-M-2-X16-V2-Threadripper/dp/B07NQBQB6Z/ref=pd_lpo_147_t_1/131-3396297-4759013?_encoding=UTF8&pd_rd_i=B07NQBQB6Z&pd_rd_r=4fb8f61a-7007-4d60-8491-3f02b8c1bfb4&pd_rd_w=txsGv&pd_rd_wg=f9tpW&pf_rd_p=16b28406-aa34-451d-8a2e-b3930ada000c&pf_rd_r=WG7V1915YQ19DZC2CSQ2&psc=1&refRID=WG7V1915YQ19DZC2CSQ2

 

This thing needs all 16 lanes to present 4 X4 NVME drives; if you have that thing in there, it seems like you're totally out of lanes before you put a video card in with Ryzen?  Do you have to put your video card in the chipset slot to make things work if you install one of these things?

 

Does anyone make a PCI4 X4 card that has 4 NVME slots on it (like the board I showed)?  Seems like that would be a really nice thing to have, with 4 NVME PCI 3 drives on it, you'd be at 1/2 the bandwidth, but still, it would be ripping fast.  I assume a NVME drive can work (somehow) on an X1 slot because it's part of the PCI standard, right?

[WhitetailAni]

3 minutes ago, Overtaxed said:

Makes sense, thanks for the details.  Let me ask something though, how on earth does someone use a card like this:

https://www.amazon.com/ASUS-M-2-X16-V2-Threadripper/dp/B07NQBQB6Z/ref=pd_lpo_147_t_1/131-3396297-4759013?_encoding=UTF8&pd_rd_i=B07NQBQB6Z&pd_rd_r=4fb8f61a-7007-4d60-8491-3f02b8c1bfb4&pd_rd_w=txsGv&pd_rd_wg=f9tpW&pf_rd_p=16b28406-aa34-451d-8a2e-b3930ada000c&pf_rd_r=WG7V1915YQ19DZC2CSQ2&psc=1&refRID=WG7V1915YQ19DZC2CSQ2

 

This thing needs all 16 lanes to present 4 X4 NVME drives; if you have that thing in there, it seems like you're totally out of lanes before you put a video card in with Ryzen?  Do you have to put your video card in the chipset slot to make things work if you install one of these things?

 

Does anyone make a PCI4 X4 card that has 4 NVME slots on it (like the board I showed)?  Seems like that would be a really nice thing to have, with 4 NVME PCI 3 drives on it, you'd be at 1/2 the bandwidth, but still, it would be ripping fast.  I assume a NVME drive can work (somehow) on an X1 slot because it's part of the PCI standard, right?

Threadrippers.

Threadrippers (at least Threadripper 3000) have 88 PCIe 4.0 lanes.

88 - 16 (GPU) - 16 (Hyper M.2) = 56 lanes left over. Plenty for 5 x4 NVMe drives on the motherboard (56 - 16 = 40) plus extra slots on the motherboard (40 - 16 - 16 = 8 lanes) and then those leftover lanes can go to the chipset.