PCIe Lane management

[Kuruderu]

I am in a... peculiar, situation. i have plenty of PCIe lanes for my usecase. But even so, i can't run my graphics card at the full 16x.

The reason for that is that i have 2 PCIe SSD's...

1 Intel 900P 

and

1 Intel 905P

 

Both are 4x drives. but mounting them in the 16x/8x lane slots in my motherboard causes my system to reserve all 8 lanes for the drive. despite it needing only 4... thus i am "wasting" 8 lanes... that could have been used for my GPU.

 

I would really love for Motherboard Manufacturers to provide us the option to manage our lane assignments a bit. so that i could limit specific slots to 4x and lock another at 16x. the traces are alle there. It should really "only" be a matter of making the proper interface in the Bios for instance... 

Edited March 6, 2023 by Kuruderu

[manikyath]

the two x16 physical slots usually share 16 lanes, and all other lanes are dedicated to other things.

 

it would be prohibitively expensive to let you 'arbitrarily' allocate lanes, to the point they may as well just carpet bomb the board with PLX chips and cover it in 16x slots. so they just tend towards the most commonly used options.. which is why just about every motherboard has the same slot loadout these days.

[RONOTHAN##]

So it would help to know what motherboard you were referring to for this for how to manage PCIe lanes. That said, assuming this is a standard consumer platform with 16 general purpose PCIe lanes from the CPU, the reasoning behind this is pretty straightforward. 

 

PCIe lane assignments for device from a technical limitation can only be given in powers of 2 on consumer platforms, so a device can have 1, 2, 4, 8, or 16 lanes dedicated to them (IIRC some server stuff can do 24 lanes, but that's the exception, not the rule). If you have 2 PCIe slots with PCIe switches, that means that if the second PCIe slot is populated at all, it has to take the full 8 lanes from the first slot, even if you are only trying to use a 1x PCIe add in card.  There's no way to take a single lane from the GPU to put towards the second slot. If you wanted to have it switch between chipset lanes and CPU lanes, that would let you do the x16/x4 PCIe lane assignment you're referring to, but the problem with that is it also gets super expensive because PCIe switches, especially Gen 5 ones found on current generation motherboards, are incredibly expensive in addition to all the added costs associated with routing those PCIe lanes and possibly the need to add even more PCB layers to accommodate them. 

 

12 minutes ago, Kuruderu said:

so that i could limit specific slots to 4x and lock another at 16x. the traces are alle there. It should really "only" be a matter of making the proper interface in the Bios for instance... 

No, that's not how it works. On a board with bifurcated PCIe slots, they share 16 lanes total, so there's no way to run it at x16/x4, there would be 4 PCIe lanes coming out of thin air, and x12 is not a supported PCIe lane assignment. 

[Kuruderu]

MY board is a ROG Crosshair VIII Dark Hero. 

This is the listed PCIe capabilites:

CPU:

2 x PCIe 4.0 x16 (x16 or dual x8)

2 x PCIe 3.0 x16 (x16 or dual x8)

1 x PCIe 3.0 x16 (x8 mode)

 

AMD X570 chipset

1 x PCIe 4.0 x16

1 x PCIe 4.0 x1

 

my CPU is a 5950x that supports 24 PCIe lanes on it's own.

Edited March 6, 2023 by Kuruderu

[Kuruderu]

4 minutes ago, manikyath said:

the two x16 physical slots usually share 16 lanes, and all other lanes are dedicated to other things.

 

it would be prohibitively expensive to let you 'arbitrarily' allocate lanes, to the point they may as well just carpet bomb the board with PLX chips and cover it in 16x slots. so they just tend towards the most commonly used options.. which is why just about every motherboard has the same slot loadout these days.

You are missing my point. I don't need more lanes on the board. i just need to be able to control the allocation of resources between the lanes that _are_ present.

[RONOTHAN##]

2 minutes ago, Kuruderu said:

my CPU is a 5950x that supports 24 PCIe lanes on it's own.

Not really. It has 4 lanes dedicated to the chipset, so that brings you down to 20 lanes, and 4 lanes are dedicated to the top M.2 slot with board vendors not allowed to put them anywhere else. The top two x16 slots can only use a combination of 16 PCIe lanes, getting any more lanes into likely add $300-400 to the cost of the motherboard alone for a feature only a handful of people would even notice, and the people who would notice are also the same people who should be on Threadripper/Sapphire Rapids. 

 

6 minutes ago, Kuruderu said:

This is the listed PCIe capabilites:

CPU:

2 x PCIe 4.0 x16 (x16 or dual x8)

2 x PCIe 3.0 x16 (x16 or dual x8)

1 x PCIe 3.0 x16 (x8 mode)

 

AMD X570 chipset

1 x PCIe 4.0 x16

1 x PCIe 4.0 x1

You forgot to copy a very important part of that, those CPU lane assignments are for Ryzen 5000/3000 CPUs, Ryzen 5000G/4000(G)/2000 CPUs, and Ryzen 3000G/2000G CPUs respectively. It doesn't have 5 x16 slots like it sounds like it's saying. 

 

8 minutes ago, Kuruderu said:

i just need to be able to control the allocation of resources between the lanes that _are_ present.

And that would require more PCIe switches than what the board has currently. 

[Kuruderu]

3 minutes ago, RONOTHAN## said:

Not really. It has 4 lanes dedicated to the chipset, so that brings you down to 20 lanes, and 4 lanes are dedicated to the top M.2 slot with board vendors not allowed to put them anywhere else. The top two x16 slots can only use a combination of 16 PCIe lanes, getting any more lanes into likely add $300-400 to the cost of the motherboard alone for a feature only a handful of people would even notice, and the people who would notice are also the same people who should be on Threadripper/Sapphire Rapids. 

 

You forgot to copy a very important part of that, those CPU lane assignments are for Ryzen 5000/3000 CPUs, Ryzen 5000G/4000(G)/2000 CPUs, and Ryzen 3000G/2000G CPUs respectively. It doesn't have 5 x16 slots like it sounds like it's saying. 

 

And that would require more PCIe switches than what the board has currently. 

I am fully aware that the current hardware is not able acomplish what i am suggesting. I was merely wondering if it would be feasable to do for board manufacturers if they so desired. Maybe as an actual additional feature on their "premium" selection of boards. 

[manikyath]

9 minutes ago, Kuruderu said:

You are missing my point. I don't need more lanes on the board. i just need to be able to control the allocation of resources between the lanes that _are_ present.

that is the point. allocating lanes isnt some voodoo magic, things need to be routed to places to make that happen, there need to be high speed busswitches, signal integrity of lanes going all over the place.. essentially board design nightmare.

 

by the time you're investing ALL that work into designing a board for some niche feature you may as well invest in some PLX chips, because that ends up in a more versatile product that's easier to manufacture, which offsets a LOT of the cost of the PLX chips.

 

so - the reason you cant, like i said, is because what you're asking is more difficult (and because of that most likely more expensive) than the more high end solution. even a PLX chip that gives you 4 propper x16 slots (or 8 x8, or 16 x4.. etc or any combination thereof) isnt *that* expensive that it would make financial sense sense to instead design a spaghetti of bus logic to route lanes where the user wants them to be.

[Blue4130]

18 minutes ago, Kuruderu said:

I am fully aware that the current hardware is not able acomplish what i am suggesting. I was merely wondering if it would be feasable to do for board manufacturers if they so desired. Maybe as an actual additional feature on their "premium" selection of boards. 

They do have premium boards with more flexibility. It's called threadripper pro/epyc. It doesn't work how you are wanting, as other have shown, but it allows you to do what you want. There already is enough market segmentation.

[Kuruderu]

2 minutes ago, Blue4130 said:

They do have premium boards with more flexibility. It's called threadripper pro/epyc. It doesn't work how you are wanting, as other have shown, but it allows you to do what you want. There already is enough market segmentation.

Retail Threadripper Pro are hard to come by. And Epyc is a bit "overkill" for a standard desktop  but yes. These platforms have _some_ of the features i am talking about.

[RONOTHAN##]

11 minutes ago, manikyath said:

even a PLX chip that gives you 4 propper x16 slots (or 8 x8, or 16 x4.. etc or any combination thereof) isnt *that* expensive that it would make financial sense sense to instead design a spaghetti of bus logic to route lanes where the user wants them to be.

Well, not really, plx chips are super expensive nowadays, something like $300+ at bulk pricing for the x32 gen 3 version. Compared to ~$40 at retail for a PCIe Gen 4x4 switch with about the same amount of spaghetti routing, the pricing will likely come out still ahead to just use switches. 

 

There's a reason they just kinda disappeared from motherboard design around Z370, they became double the price of the boards that they'd go on, up from about $100 for a 32 lane switch. 

[Radium_Angel]

3 minutes ago, Kuruderu said:

Retail Threadripper Pro are hard to come by. And Epyc is a bit "overkill" for a standard desktop  but yes. These platforms have _some_ of the features i am talking about.

Older Xeons work...40 PCIe lanes...

[Kuruderu]

10 minutes ago, Radium_Angel said:

Older Xeons work...40 PCIe lanes...

True. Case in point "Older Xeons"  not exactly what i am looking for

[manikyath]

1 minute ago, RONOTHAN## said:

Well, not really, plx chips are super expensive nowadays, something like $300+ at bulk pricing for the x32 gen 3 version. Compared to ~$40 at retail for a PCIe Gen 4x4 switch with about the same amount of spaghetti routing, the pricing will likely come out still ahead to just use switches. 

 

There's a reason they just kinda disappeared from motherboard design around Z370, they became double the price of the boards that they'd go on, up from about $100 for a 32 lane switch. 

i've been hunting for expansion boards for my server a while back, the actual "PLX" brand ones are stupid expensive - yes. but they arent the only brand out there. i'm using "PLX" in the "kleenex" sense of the word.

 

but to a degree the cost of a "PCIe switch chipset" is reflected in the amount of busswitches, board space, and design needed to do what OP wants to do. (take N CPU lanes + M chipset lanes, and spread them across the slots in a user-configurable way)

 

they're both prohibitively expensive for a mainstream board, but i'd say that they're within the same order of magnitude "prohibitively expensive".

 

14 minutes ago, Kuruderu said:

Retail Threadripper Pro are hard to come by. And Epyc is a bit "overkill" for a standard desktop  but yes. These platforms have _some_ of the features i am talking about.

the reason TR and epyc are so configurable, is because they literally have lanes to piss away. in a sense, TR has even more I/O to piss away than epyc (because TR actually *has* a chipset), and on epyc it's not unlikely to see motherboard manufacturers to just use a full x16 segment to drive vareous random things on the motherboard, and you still have a stack of full length slots.

 

but really.. if you're running out of lanes on epyc, that's a you problem

[Kuruderu]

3 minutes ago, manikyath said:

i've been hunting for expansion boards for my server a while back, the actual "PLX" brand ones are stupid expensive - yes. but they arent the only brand out there. i'm using "PLX" in the "kleenex" sense of the word.

 

but to a degree the cost of a "PCIe switch chipset" is reflected in the amount of busswitches, board space, and design needed to do what OP wants to do. (take N CPU lanes + M chipset lanes, and spread them across the slots in a user-configurable way)

 

they're both prohibitively expensive for a mainstream board, but i'd say that they're within the same order of magnitude "prohibitively expensive".

 

the reason TR and epyc are so configurable, is because they literally have lanes to piss away. in a sense, TR has even more I/O to piss away than epyc (because TR actually *has* a chipset), and on epyc it's not unlikely to see motherboard manufacturers to just use a full x16 segment to drive vareous random things on the motherboard, and you still have a stack of full length slots.

 

but really.. if you're running out of lanes on epyc, that's a you problem

yep. TR and epyc indeed has lanes to piss away  But as i, somewhat ineptly, tried to convey was to get a sense of why/how it could be managed on more "mainstream" hardware... albeit in the "premium segment" naturally

[manikyath]

Just now, Kuruderu said:

, tried to coney was to get a sense of why/how it could be managed on more "mainstream" hardware... albeit in the "premium segment" naturally

and what i'm trying to convey, is that it is not feasible.

[Radium_Angel]

3 minutes ago, Kuruderu said:

True. Case in point "Older Xeons"  not exactly what i am looking for

The v4 series (Broadwell I think) are still highly capable. I'm typing this on an e5-2690 v4 (14 cores, 28 threads) that supports absolutely stupid amounts of RAM, and those lovely 40 PCIe lanes.

I added a modest GPU (GTX 1060 6gb model) and it plays Cyberpunk on all high settings just smooth as butter

I paid...40$ for the chip.

[Blue4130]

33 minutes ago, Kuruderu said:

Retail Threadripper Pro are hard to come by. And Epyc is a bit "overkill" for a standard desktop  but yes. These platforms have _some_ of the features i am talking about.

Not really hard to come by. Expensive? Sure.