PCIe Lanes Confusion

[Alpha297]

I wasn't sure if this should be posted in Workbench or here, but I opted for here since it's not a build specific question, but trying to clear up some confusion I've got on how PCIe lanes are used.

 

For this hypothetical lets use an i-5 7500 which has 16 PCIe lanes per ARK.

For our hypothetical chipset, lets use the Z270 which has 24 PCIe lanes per ARK

 

Now with that established, here is my understanding of what that means:

 

The CPU and the chip set lanes are separate making a total of 40 available lanes for this potential setup.  The 16 CPU lanes, according to the chipset specifications can be used in 1x16 or 2x8 or 1x8+2x4. Meaning one graphics card at 16x, Dual cards at 8x each, and so forth.  So you could have 1 graphics card running at 16x and the remaining 24 lanes on the chipset for devices such as M.2 cards, or whatever else you wanted to plug in.

 

I'm pretty confident on that part from what I've read, but here's where it gets fuzzy.

 

1. If you put a two graphics card in, what keeps one from running 16x off the CPU lanes, and one from running 16x off the chipset lanes, leaving another 8 lanes unused.   (I understand that there's no reason to really do this, that two cards running at 8x is perfectly legit, but I'm asking from a theoretical perspective.)
2. What determines if a given slot is running off the CPU lanes or the chipset lanes?
3. What would happen if you say, plugged a 16x something not GPU card into a CPU lane slot, and your GPU in another 16x slot on your board?

 

I hope I asked in a coherent way.  Thanks! 

 

 

[SpaceGhostC2C]

1 minute ago, Alpha297 said:

 

1. If you put a two graphics card in, what keeps one from running 16x off the CPU lanes, and one from running 16x off the chipset lanes, leaving another 8 lanes unused.   (I understand that there's no reason to really do this, that two cards running at 8x is perfectly legit, but I'm asking from a theoretical perspective.)
2. What determines if a given slot is running off the CPU lanes o

The motherboard traces. The lanes are physically traces on the motherboard connecting devices. Think of it like wires, or actual road lanes. Some slots are routed directly to the CPU, while others go via the chipset.

Also, bear in mind that the chipset iotself has a particular connection with the CPU, with a given bandwidth. Typically, that connection has far less bandwidth than 24 PCIe lanes, so if you were to use all 24 lanes simultaneously, and with devices capable of using all of the available bandwidth, they couldn't all work at full speed at the same time (it would be really hard to make such a setup, but it would matter for GPUs).

 

1 minute ago, Alpha297 said:

1. What would happen if you say, plugged a 16x something not GPU card into a CPU lane slot, and your GPU in another 16x slot on your board?

I'm guessing your GPU would still work, but probably at something like 4x speed, since that's more or less the bandwidth between CPU and chipset as far as I understand.

[tp95112]

1. Manufacture and bios software. Also some board will have both 16x slots run from the cpu. 

2. Manufacturer determines which goes to which. Usually the top goes to the cpu

3. Then what ever goes to the first will use the cpu and gpu will use chipset. They are physically wired to chipset or cpu

[Alpha297]

Thanks both.  

 

So in this theoretical example, you could say.. multi GPU at 2x8 with full capabilities, since the CPU is designed for that. , But if you tried the concept in my option 1, using two 16x cards, it would work but not well?  You'd run into the issue of the chipset lanes running at different overall speed then the CPU lanes?

[Bananasplit_00]

1: not much beyond that how its wired and latency issues seeing as data has to go through the chipset to the CPU in the end.

2: the traces on the motherboard, like the physical copper traces in the motherboard

3: preformance would be wonky because of latency most likely, your motherboard would have to be pretty stripped of features and other expansion. you would probably just have minimal I/O and no M.2, SATA express and so on and no aditional PCIe slots either. this would probably end up with lower preformance then just running 8/8 because of latency and stuttering would most likely be the main problem. no graphics card really saturates 8x 3.0 anyway so there is no reason to do this either way

[NumLock21]

Cheap z270 boards, all 16 cpu lanes goes to the 1st x16 slot

 Everything else is from the chipset.

[Electronics Wizardy]

39 minutes ago, Alpha297 said:

Thanks both.  

 

So in this theoretical example, you could say.. multi GPU at 2x8 with full capabilities, since the CPU is designed for that. , But if you tried the concept in my option 1, using two 16x cards, it would work but not well?  You'd run into the issue of the chipset lanes running at different overall speed then the CPU lanes?

you can't do 16x over the chipset, its 4x max on the chipset on any device, you can run 4 x 4x but no 1x 16x on the chipset.

[Alpha297]

1 minute ago, Electronics Wizardy said:

you can't do 16x over the chipset, its 4x max on the chipset on any device, you can run 4 x 4x but no 1x 16x on the chipset.

So why is that?  Other responses would have made me think you can but it'd be slowed up by having to go through the chipset and back around.

[Electronics Wizardy]

6 minutes ago, Alpha297 said:

So why is that?  Other responses would have made me think you can but it'd be slowed up by having to go through the chipset and back around.

limitation of the chipset. Also no reason to. The bandwidth from the cpu is about the same as pcie 4x.

[SpaceGhostC2C]

13 minutes ago, Alpha297 said:

So why is that?  Other responses would have made me think you can but it'd be slowed up by having to go through the chipset and back around.

Mm.. not really, we did say the communication between CPU and chipset is the bottleneck, and that its bandwidth it's like 4x 

 

Basically, you have a 16-lane going from the first PCIex16 to the CPU, and then you have highways of varying sizes going to the chipset, but only a 4-lane road going from the chipset to the CPU. So, whatever amount of traffic you manage to send till the chipset, it still has to make the final mile through a 4-lane road.

[Alpha297]

1 minute ago, SpaceGhostC2C said:

Mm.. not really, we did say the communication between CPU and chipset is the bottleneck, and that its bandwidth it's like 4x 

 

Basically, you have a 16-lane going from the first PCIex16 to the CPU, and then you have highways of varying sizes going to the chipset, but only a 4-lane road going from the chipset to the CPU. So, whatever amount of traffic you manage to send till the chipset, it still has to make the final mile through a 4-lane road.

I must have misunderstood then.  But this is why I poked my head out here to ask questions

[Alpha297]

So just to make sure I understand the subsequent clarifications.  The only way to get "true" 2x16 graphics card throughput, (understanding that theres no realistic need to do that) would be to have a CPU that supports at least 32 PCIe lanes such as the  i7-6850K with a whopping 40 PCIe lanes?

 

That's a lot of lanes.    I hadn't looked at any of the X series being well out of my price range for my build so.. yeah I have no idea what you'd use that many lanes for.  

 

[Technous285]

There's not much performance difference between running GPU's at 2x16 PCIe 3.0 & 2x8 PCIe 3.0, so it's not really worth shelling out for a chip on X99/LGA2011v3 unless you're going to be running 3-4 video cards (eg: 3-way SLI on 5820K or 6800K with their 28 lanes. or 4-way SLI on 5930K/5960X or 6850K/6900K/6950X with their 40 lanes).

[SpaceGhostC2C]

13 hours ago, Alpha297 said:

So just to make sure I understand the subsequent clarifications.  The only way to get "true" 2x16 graphics card throughput, (understanding that theres no realistic need to do that) would be to have a CPU that supports at least 32 PCIe lanes such as the  i7-6850K with a whopping 40 PCIe lanes?

 

That's a lot of lanes.    I hadn't looked at any of the X series being well out of my price range for my build so.. yeah I have no idea what you'd use that many lanes for.  

 

Indeed. I mean, there used to be more 16x 16x alternatives with PCIe 2.0. But then again, 16x PCIe 2.0 is the same bandwidth as 8x PCIe 3.0, so dual 8x is still as fast.

You would need dual 1080ti or Titan to start noticing a limit, though.

[Alpha297]

Thanks for the info all.  I feel a lot more confident in the terminology and what it all means as it relates to my questions.