PCIe lanes

[Bubben]

I'd like to get the whole lane thing clarified as I don't have a whole lot of knowledge on the subject.

 

So there are a total 16 lanes connected directly to the cpu in most modern systems, and these 16 lanes are divided between the 16x slots depending on how many of them you have occupied? In that case what about the remaining 1x slots and general I/O. Are there more lanes provided to them indirectly at the cost of latency, or do they have to share the same 16 lanes as well?

 

I appreciate any insight.

[Godlygamer23]

PCI-E 1x slots are run by the chipset. Same with SATA and USB.

[target39]

I think it depends on the CPU. I don't claim to know much about this, but this is what I think. Some LGA2011 CPUs for instance, support 40 PCI-E lanes, meaning they can run two way SLI/Crossfire with 16x bandwidth on each card taking up 32 total lanes. 8 lanes will have been left for other components such as PCI-E SSDs, sound cards, etc. This is why it splits 4 way SLI/Crossfire into 16x/8x/8x/8x instead of 16x/16x/16x/16x because that would exceed 40 lanes. Obviously you have boards with special embedded chips such as the ASRock X79 Extreme11 which allows you to run 16x/16x/16x/16x, but this is usually the case. Latency is based on the generation of PCI-E they use and the length and resistance of the conductor between the components. This means you usually wouldn't get PCI-E 3.0 with a 50 foot 50AWG bus PCI-E extender cable without the use of other components. If I am wrong, don't be afraid to correct me as I am trying to understand this as well. 

[target39]

PCI-E 1x slots are run by the chipset. Same with SATA and USB.

Some motherboards actually have chipset controlled SATA combined with a Marvell SATA controller.

[Godlygamer23]

Some motherboards actually have chipset controlled SATA combined with a Marvell SATA controller.

Yes. My motherboard has a Marvell SATA controller, but the chipset still controls most of the SATA ports and USB ports(unless you have some sort of extra USB 3.0 controller, like my board) and controls the PCI-E 1x slots.

[Bubben]

PCI-E 1x slots are run by the chipset. Same with SATA and USB.

Should I take it then that the bandwidth provided through the 16x slots is separate from the rest of system, and that nothing else can intrude on it?

[Bubben]

I think it depends on the CPU. I don't claim to know much about this, but this is what I think. Some LGA2011 CPUs for instance, support 40 PCI-E lanes, meaning they can run two way SLI/Crossfire with 16x bandwidth on each card taking up 32 total lanes. 8 lanes will have been left for other components such as PCI-E SSDs, sound cards, etc. This is why it splits 4 way SLI/Crossfire into 16x/8x/8x/8x instead of 16x/16x/16x/16x because that would exceed 40 lanes. Obviously you have boards with special embedded chips such as the ASRock X79 Extreme11 which allows you to run 16x/16x/16x/16x, but this is usually the case. Latency is based on the generation of PCI-E they use and the length and resistance of the conductor between the components. This means you usually wouldn't get PCI-E 3.0 with a 50 foot 50AWG bus PCI-E extender cable without the use of other components. If I am wrong, don't be afraid to correct me as I am trying to understand this as well. 

So you mean to say that the lga2011 systems might have been a case where the total bandwidths was shared by the entire I/O then?

[Bubben]

To answer my own question with the z77 chipset as an example.

There're a total of 20 pci-e 3.0 lanes available, each with a bandwidth of 7,88Gbit/s in both directions. 16 of these lanes are dedicated to the 16x slots and connected directly to what use to be the 'Northbridge', but the northbridge is nowadays integrated directly with the cpu. The remaining 4 lanes are sent through the DMI (where the bandwidth is converted to pci-e 2.0 lanes?) to what more or less used to be the 'Southbridge' but is now more commonly referred to as the actual chipset. At the chipset the bandwidth is divided as needed in between remaining pci slots, sata and general I/O.

Or at least that's how I understood it, so don't necessarily take my word for it.

[Godlygamer23]

Should I take it then that the bandwidth provided through the 16x slots is separate from the rest of system, and that nothing else can intrude on it?

Yes. Generally, if you have 16 lanes, and add two cards into 16x length slots, they'll be split into 8x/8x. But if you have 20 lanes, that may apply to an x4 slot where the extra 4 lanes will be for that slot. Anything that's 1x, SATA, or USB is controlled by the chipset generally, unless you have a third party controller, either PCI-E, SATA, or USB.

[Bubben]

Yes. Generally, if you have 16 lanes, and add two cards into 16x length slots, they'll be split into 8x/8x. But if you have 20 lanes, that may apply to an x4 slot where the extra 4 lanes will be for that slot. Anything that's 1x, SATA, or USB is controlled by the chipset generally, unless you have a third party controller, either PCI-E, SATA, or USB.

If I've understood it correctly then it's the remaining lanes that are allocated to the chipset, and through there to the remaining 1x, sata, I/O etc. Any third party controller would have to share that bandwidth as well.

[Godlygamer23]

If I've understood it correctly then it's the remaining lanes that are allocated to the chipset, and truth there to the remaining 1x, sata, I/O etc. Any third party controller would have to share that bandwidth as well.

No. The third party controller doesn't share the bandwidth.

[Bubben]

No. The third party controller doesn't share the bandwidth.

Then where dose it get the bandwidth from?

[Godlygamer23]

Then where dose it get the bandwidth from?

The controller tells the slots to be at 16x, as long as the slots can run at that speed. Which is why you're able to get boards with all PCI-E slots running at 16x. However, using a third party controller introduces increased latency.

[Bubben]

The controller tells the slots to be at 16x, as long as the slots can run at that speed. Which is why you're able to get boards with all PCI-E slots running at 16x. However, using a third party controller introduces increased latency.

What about a board that doesn't have a pci-e switch? Or dose something like a ASMedia sata controller implement a switch of some kind?

[Godlygamer23]

What about a board that doesn't have a pci-e switch? Or dose something like a ASMedia sata controller implement a switch of some kind?

I'm not sure I fully understand what you're saying.

[Bubben]

I'm not sure I fully understand what you're saying.

A board with only 16 lanes dedicated to the 16x slots can only run all slots at full 16x speeds if there's a pci-e switch like a PLX involved. These chips don't actually increase the bandwidths as such (not possible), instead what they basically do is multiplexing (see bellow) which allows multiple pci-e cards to share the same lanes, this is where the latency comes in.

 

 

As I said before; any third party controller present on the board would have to share the lanes available to the chipset, and after a bit of googling it dose indeed seem like at least some ASMedia controllers and the like implement pci-e switching, but not on the 16 dedicated lanes

[Godlygamer23]

A board with only 16 lanes dedicated to the 16x slots can only run all slots at full 16x speeds if there's a pci-e switch like a PLX involved. These chips don't actually increase the bandwidths as such (not possible), instead what they basically do is multiplexing (see bellow) which allows multiple pci-e cards to share the same lanes, this is where the latency comes in.

As I said before; any third party controller present on the board would have to share the lanes available to the chipset, and after a bit of googling it dose indeed seem like at least some ASMedia controllers and the like implement pci-e switching, but not on the 16 dedicated lanes

Was not aware of that.

[Bubben]

Was not aware of that.

You live and learn.

[knightshade]

Ok, I've got a related question...

 

I'm building my first new system in a while... what I need is the following:

 

1 16x PCIe 3.0 slot (running at 16x) for a video card

1 4x PCIe slot (can be 2.0 or 3.0 don't care) for a RAID card

1 1x PCIe slot for a sound card.

 

A lot of the boards I'm looking at are either dropping the 16x slot down to 8x if you put anything in any slot besides a 1x slot... and/or disabling all their 1x slots if you put anything in any slot but the first 16x video slot.

 

So is the above even possible? (ideally without using a PLX chip as I've read not great things about em)

 

if so what Z87 motherboards can do it?

 

if not I suppose I'll just have to skip the discrete audio card.

 

Much thanks in advance!

[Godlygamer23]

Ok, I've got a related question...

 

I'm building my first new system in a while... what I need is the following:

 

1 16x PCIe 3.0 slot (running at 16x) for a video card

1 4x PCIe slot (can be 2.0 or 3.0 don't care) for a RAID card

1 1x PCIe slot for a sound card.

 

A lot of the boards I'm looking at are either dropping the 16x slot down to 8x if you put anything in any slot besides a 1x slot... and/or disabling all their 1x slots if you put anything in any slot but the first 16x video slot.

 

So is the above even possible? (ideally without using a PLX chip as I've read not great things about em)

 

if so what Z87 motherboards can do it?

 

if not I suppose I'll just have to skip the discrete audio card.

 

Much thanks in advance!

Yes. You can do that. You don't need a Haswell chip that can do it.

[Godlygamer23]

Here's a Z87 board that can do that from ASUS. http://www.asus.com/us/Motherboards/Z87DELUXE/

[Bubben]

Ok, I've got a related question...

 

I'm building my first new system in a while... what I need is the following:

 

1 16x PCIe 3.0 slot (running at 16x) for a video card

1 4x PCIe slot (can be 2.0 or 3.0 don't care) for a RAID card

1 1x PCIe slot for a sound card.

 

A lot of the boards I'm looking at are either dropping the 16x slot down to 8x if you put anything in any slot besides a 1x slot... and/or disabling all their 1x slots if you put anything in any slot but the first 16x video slot.

 

So is the above even possible? (ideally without using a PLX chip as I've read not great things about em)

 

if so what Z87 motherboards can do it?

 

if not I suppose I'll just have to skip the discrete audio card.

 

Much thanks in advance!

The Asus z87 Pro and up is able to do 16x pci-e 3.0 + 4x pci-e 2.0 + 1x. The bottom 16x slot is wired for 4x and it will disable all but one of the 1x slots if you use it. That said, you will be pushing it and the x79 chipset would be better suited for it, with double the number of lanes.

 

(Edit) The deluxe uses PLX btw.

[knightshade]

The Asus z87 Pro and up is able to do 16x pci-e 3.0 + 4x pci-e 2.0 + 1x. The bottom 16x slot is weird for 4x and it will disable all but one of the 1x slots if you use it. That said, you will be pushing it and there are chipsets better suited for it, like x79 which will give you 40 lanes total.

 

(Edit) The deluxe uses PLX btw.

 

Thanks for the reply...  If I'm understanding, the board has 16 PCIe 3 lanes which would all go to video in the PCIe 16x_1 slot.... the 16x_2 slot would sit empty...then PCIe 2.0 from the chipset has 8 lanes... with 4 going to the 16x_3 slot to run the RAID at 4x... 1 going to the 1x_1 slot that still works, the 1x_2,3,4 slots disabled... and 3 PCIe 2.0 lanes left for other chipset stuff  (USB and whatnot?).... 

 

If that's correct, Can you elaborate a bit on what "pushing it" entails in this case?

 

I'm reluctant to consider the x79 as it pushes me back 2 gens on CPU to Sandy Bridge (albeit E), and nearly doubles by CPU/motherboard price ($560ish for a Sandy E versus a $330ish 4770k plus the pricier X79 board)... which seems a large difference to pay if the Z87 Pro will work for me.

[Bubben]

knightshade, on 07 Jun 2013 - 04:18 AM, said:

Thanks for the reply... If I'm understanding, the board has 16 PCIe 3 lanes which would all go to video in the PCIe 16x_1 slot.... the 16x_2 slot would sit empty...then PCIe 2.0 from the chipset has 8 lanes... with 4 going to the 16x_3 slot to run the RAID at 4x... 1 going to the 1x_1 slot that still works, the 1x_2,3,4 slots disabled... and 3 PCIe 2.0 lanes left for other chipset stuff (USB and whatnot?)....

If that's correct, Can you elaborate a bit on what "pushing it" entails in this case?

I'm reluctant to consider the x79 as it pushes me back 2 gens on CPU to Sandy Bridge (albeit E), and nearly doubles by CPU/motherboard price ($560ish for a Sandy E versus a $330ish 4770k plus the pricier X79 board)... which seems a large difference to pay if the Z87 Pro will work for me.

If you have the very top and the bottom 16x slots occupied then the top 1x slot will be the one that's left working. The bottom 16x and top 1x would run of the effective 5 pci-e 2.0 lanes provided though the chipset.

The reason why I say that you'll be pushing it is because there's only so much bandwidths available. In fact if I'm not mistaken you might end up saturating the 4x raid card, seeing as it'll be taking up 16Gbits of the 20Gbits available, and that's before you factor in anything else you might have connected.

If I might ask you though, have you seen any benchmarks comparing pci-e 16x vs 8x? You really wouldn't be missing anything on a single gpu setup.

[Godlygamer23]

If you have the very top and the bottom 16x slots occupied then the top 1x slot will be the one that's left working.

 

The reason why I say that you'll be pushing it is because there's only so much bandwidths available. In fact if I'm not mistaken you might end up saturating the 4x raid card, seeing as it'll be taking up 16Gbits of the 20Gbits available, and that's before you factor in anything else you might have connected.

 

If I might ask you though, have you seen any benchmarks comparing pci-e 16x vs 8x? You really wont be missing anything on a single gpu setup.

The 1x slot works through the chipset. With that in mind, 16 lanes+4=20.