Skylake & Haswell-E PCIe lane misconception

[AManWithPlan]

On 10/6/2015 at 5:19 AM, Sakkura said:

As for the Core i7-5820K, it offers the well-known 28 PCIe 3.0 lanes directly from the CPU. But again, the chipset offers additional PCIe lanes. In this case, X99 is far inferior to Z170, because it only offers 8 PCIe 2.0 lanes (just like Z97).

The 5820k is a 28 lane CPU while the 6700k is a 16 lane CPU. While theoretically a full graphics card only takes 4 to 8 lanes to fully utilize its speed, software-wise you can always soft-bottleneck if you are running dual GPUs with a 6700k processor.

 

The biggest part I disagree with is the inferiority of the X99 board. I have an ASUS X99-Pro running a 5930k processor (40 lanes). This allows me to run two GPUs at 16 lanes a piece, a PCIe SSD at 4 lanes, and not bottleneck any of my other components.

 

The X99 board can handled 3 independent PCIe 3.0 (or 2.0) connections at 16x a piece, however, no Haswell-E processor has the lanes to support such a connection as this exceeds its 40 lane size. I don't believe 6000 series processors are in any better shape to handle this either.

[Sakkura]

34 minutes ago, AManWithPlan said:

The 5820k is a 28 lane CPU while the 6700k is a 16 lane CPU. While theoretically a full graphics card only takes 4 to 8 lanes to fully utilize its speed, software-wise you can always soft-bottleneck if you are running dual GPUs with a 6700k processor.

 

The biggest part I disagree with is the inferiority of the X99 board. I have an ASUS X99-Pro running a 5930k processor (40 lanes). This allows me to run two GPUs at 16 lanes a piece, a PCIe SSD at 4 lanes, and not bottleneck any of my other components.

 

The X99 board can handled 3 independent PCIe 3.0 (or 2.0) connections at 16x a piece, however, no Haswell-E processor has the lanes to support such a connection as this exceeds its 40 lane size. I don't believe 6000 series processors are in any better shape to handle this either.

I was comparing the chipsets. The reason you can run all those things is that the CPU provides lots of PCIe lanes, it's not because of the chipset. As I explained, the X99 chipset is strictly inferior to the Z170 chipset (and now also Z270). The motherboard is basically just a conduit for the connectivity enabled by the CPU and the chipset; and as it happens, the vast majority of the PCIe capability of your platform stems from the CPU, not the chipset.

[AManWithPlan]

27 minutes ago, Sakkura said:

I was comparing the chipsets. The reason you can run all those things is that the CPU provides lots of PCIe lanes, it's not because of the chipset. As I explained, the X99 chipset is strictly inferior to the Z170 chipset (and now also Z270). The motherboard is basically just a conduit for the connectivity enabled by the CPU and the chipset; and as it happens, the vast majority of the PCIe capability of your platform stems from the CPU, not the chipset.

Unless I am really old here, what does a chipset or PCH have to do with PCIe lanes? Aren't PCIe, by definition, routed directly to CPU?

[Sakkura]

13 minutes ago, AManWithPlan said:

Unless I am really old here, what does a chipset or PCH have to do with PCIe lanes? Aren't PCIe, by definition, routed directly to CPU?

The chipset provides a number of PCIe lanes separately from the ones the CPU offers directly.

 

On an X99 system, you get 8 PCIe 2.0 lanes from the chipset. On a Z170 or Z270 system, you get up to 20 PCIe 3.0 lanes from the chipset (though it depends how many of the HSIO ports the motherboard manufacturer decides to dedicate to PCIe).

 

The chipset acts as a bridge chip here, and you can also have other controllers that perform a similar function. PCIe lanes thus do not have to go straight to the CPU.

[zeroyon]

Skylake/Kabylake has a lot of disadvantages with regards to PCIe because if you try and use any of the M.2 slots on almost any of the Skylake or Kabylake MOBOs, it will disable PCIe slots or even sometimes several SATA ports.

 

I got my Extreme11 because of the two PLX chips (providing 64 PCIe lanes) and LSI SAS controller, so I can run my many HDDs and have room to expand to many NVMe drives over PCIe in the future without crippling other ports or slots in my PC.

 

If you only have a few HDDs or plan on never going to SLI/Crossfire, it shouldn't be an issue though.

[The Benjamins]

@Sakkura

Isn't the Skylake-X the same were it has 28/44 from the CPU and "24" from the chipset but the chipset only has a DMI 3.0 link at 4x.

 

Quote

For most practical definitions of the Basin Falls platform, the X299 chipset is the heart. X299 supports the new processors, and like the Z170 and Z270 counterparts on the mainstream consumer line, is basically a big PCIe switch. One of the issues with the older X99 chipset was its limited capabilities, and inability to drive many PCIe devices – this changes with the big switch mentality on X299. For the DMI 3.0 link going into the chipset (basically a PCIe 3.0 x4), the chipset has access to up to 24 PCIe 3.0 lanes for network controllers, RAID controllers, USB 3.1 controllers, Thunderbolt controllers, SATA controllers, 10GbE controllers, audio cards, more PCIe slot support, special controllers, accelerators, and anything else that requires PCIe lanes in either an x4, x2 or x1 link. The total uplink is limited by the DMI 3.0 link, but there will be very few situations where this is saturated. There are a few limits to what support is available (some ports are restricted in what they can handle), and only three PCIe 3.0 x4 drives can use the in-built PCIe RAID, but this should satiate all but the most hardcore enthusiasts.

http://www.anandtech.com/show/11550/the-intel-skylakex-review-core-i9-7900x-i7-7820x-and-i7-7800x-tested/2

[Sakkura]

23 minutes ago, The Benjamins said:

@Sakkura

Isn't the Skylake-X the same were it has 28/44 from the CPU and "24" from the chipset but the chipset only has a DMI 3.0 link at 4x.

 

http://www.anandtech.com/show/11550/the-intel-skylakex-review-core-i9-7900x-i7-7820x-and-i7-7800x-tested/2

Skylake-X and X299 sort of has the best of both worlds - lots of CPU PCIe lanes like Haswell-E and Broadwell-E, lots of chipset PCIe lanes like Z170 and Z270 (but still with the DMI 3.0 limit).

[The Benjamins]

Just now, Sakkura said:

Skylake-X and X299 sort of has the best of both worlds - lots of CPU PCIe lanes like Haswell-E and Broadwell-E, lots of chipset PCIe lanes like Z170 and Z270 (but still with the DMI 3.0 limit).

ya, some people in the beginning told me x299 will be 28/44 +24 when it really isn't which puts in 20 behind Threadripper.

I watched a video going over 3 X299 boards and I think it is silly how much stuff gets disabled when different CPU's are used or when some I/O is used.

 

[This kid builds pc]

Guys, I'm from the future and AMD HAS RYZEN. Both of them are not good values, and you should buy ryzen instead.

[TheSaint]

Question regarding PCIe lanes:

 

My current setup: 

 

Asrock Z170 OC Formula Mobo (user manual here: http://asrock.pc.cdn.bitgravity.com/Manual/Z170 OC Formula.pdf) (pages 23 and 42 go over shared lanes)

6700k CPU

Nvidia 1070 GPU

m.2 PCIe Samsung 950 SSD

SATA Crucial SSD

 

Quick spec sheet of my Asrock mobo: http://www.asrock.com/mb/Intel/Z170 OC Formula/?cat=Specifications

 

I'm trying to calculate my total lane usage and see if I'm going to experience any bottleneck if I add the following devices and not remove anything listed above:

 

• 10GBe NIC card (for connecting to a NAS I'm building)
• Adding either an additional SSD or HDD via SATA
• Adding another m.2 drive

The most important thing I'm concerned about is getting the 10GBe NIC in my machine, as I can always use the NAS for storage and reduce lane usage in my desktop. 

 

I'm looking at buying two of these guys: http://www.tweaktown.com/news/58282/asus-launch-10gbe-pcie-adapter-costs-99/index.html

 

One 10gbe NIC for my desktop, and one for my upcoming FreeNAS server build. Will putting this extra 10gbe reduce the availability or speed of my other devices? My reading of my mobo manual says it won't reduce the availability of anything, but it will slow my GPU slot (currently running at 16x) to 8x once the NIC is installed, with the NIC running at 8x too. Is 8x still above what would be a bottleneck to get full performance out of both devices?

 

Thanks for any pointers!

[Stefan Payne]

On 22.3.2017 at 4:25 PM, Sakkura said:

On an X99 system, you get 8 PCIe 2.0 lanes from the chipset.

Yes

But don't forget the latency and how the chipset is connected to the CPU!

If you need the bandwith, the chipset has 8 PCIe lanes but is only connected with 4 to the CPU, its useless for higher bandwith things or things that want as low latency as you can get....

On 22.3.2017 at 4:25 PM, Sakkura said:

On a Z170 or Z270 system, you get up to 20 PCIe 3.0 lanes from the chipset (though it depends how many of the HSIO ports the motherboard manufacturer decides to dedicate to PCIe).

Well, no.

The chipset has 20 High speed PHY. But those are all shared with other things.

PCIe, S-ATA and USB3.0 all use the same PHY but internally you can switch some ports between PCIe/SATA or AFAIR PCIe/USB 3.0

 

But in the End it makes no sense to use all 20 Ports for PCIe!!

Most of them are used for S-ATA (-6) , anoher 4 can be used for another 4 USB 3.0 ports...

 

Anandtech has a nice diagram:

https://www.anandtech.com/show/9582/intel-skylake-mobile-desktop-launch-architecture-analysis/4

 

So in the end, the advantage of the Z170 isn't as big as you tell us it is. 

It is just more flexible than X99. But you also forget to mention the buttload of S-ATA connectors the X99 has.

 

On 22.3.2017 at 4:25 PM, Sakkura said:

The chipset acts as a bridge chip here, and you can also have other controllers that perform a similar function. PCIe lanes thus do not have to go straight to the CPU.

Yes, but the ones in the CPU have a lower latency, thus higher performance. And thus you want the high performance devices connected to the CPU.

Also the connection of chipset and CPU limits the overall bandwith of the chipset.

 

And here we are talking about 4 Lanes for the Z170 as well as for the X99.

 

Now what does each chipset have:
X99: 6 USB 3.0 + 10 S-ATA + 8 PCIe (2.0) makes a total of 24 PHY, though they are hard connected.

The Z170 has a total of 26 Lanes - just two more...

6 of those 26 lanes are hardwired to USB 3.0, another 4 can be either USB 3.0 or PCIe 3.0, another 6 lanes can be either S-ATA or PCIe. That's what that "up to" means...

[Stefan Payne]

On 20.6.2017 at 2:13 PM, Sakkura said:

Skylake-X and X299 sort of has the best of both worlds - lots of CPU PCIe lanes like Haswell-E and Broadwell-E, lots of chipset PCIe lanes like Z170 and Z270 (but still with the DMI 3.0 limit).

...until Treadripper comes along...

 

The CPU has 64 PCIe lanes. 

All of them. Even the rather cheap 1900X. 

Why would you want a Skylake-X anyway??

Only if you don't buy that POS, you can force Intel to change their politics...

[Sakkura]

21 minutes ago, Stefan Payne said:

Yes

But don't forget the latency and how the chipset is connected to the CPU!

If you need the bandwith, the chipset has 8 PCIe lanes but is only connected with 4 to the CPU, its useless for higher bandwith things or things that want as low latency as you can get....

Well, no.

The chipset has 20 High speed PHY. But those are all shared with other things.

PCIe, S-ATA and USB3.0 all use the same PHY but internally you can switch some ports between PCIe/SATA or AFAIR PCIe/USB 3.0

 

But in the End it makes no sense to use all 20 Ports for PCIe!!

Most of them are used for S-ATA (-6) , anoher 4 can be used for another 4 USB 3.0 ports...

 

Anandtech has a nice diagram:

https://www.anandtech.com/show/9582/intel-skylake-mobile-desktop-launch-architecture-analysis/4

 

So in the end, the advantage of the Z170 isn't as big as you tell us it is. 

It is just more flexible than X99. But you also forget to mention the buttload of S-ATA connectors the X99 has.

 

Yes, but the ones in the CPU have a lower latency, thus higher performance. And thus you want the high performance devices connected to the CPU.

Also the connection of chipset and CPU limits the overall bandwith of the chipset.

 

And here we are talking about 4 Lanes for the Z170 as well as for the X99.

 

Now what does each chipset have:
X99: 6 USB 3.0 + 10 S-ATA + 8 PCIe (2.0) makes a total of 24 PHY, though they are hard connected.

The Z170 has a total of 26 Lanes - just two more...

6 of those 26 lanes are hardwired to USB 3.0, another 4 can be either USB 3.0 or PCIe 3.0, another 6 lanes can be either S-ATA or PCIe. That's what that "up to" means...

Please read my original post before you start your ignorant trolling.

[Manuel Bachmann]

Not shure if i get this hole Lanes stuff right. Just try to understand:
If i have a GPU (on PCI-E 16 Lanes), a m.2 SSD (on PCI-E 8 Lanes) and a Soundcard (on the PCI-E 4 Lanes) means i use 28 Lanes.
But my CPU only support 16 Lanes means i lost some Performance; right??

My PC:

Corsair RM850 
Gigabyte GA-Z87X-D3H Motherboard
i7 4790K 

32GB RAM @ 2133MhZ
Quadro P4000
Kingston m.2 SSD

X-Fi Soundblaster (Creative)
 

[Bajantechnician]

25 minutes ago, Bachmann Photography said:

Not shure if i get this hole Lanes stuff right. Just try to understand:
If i have a GPU (on PCI-E 16 Lanes), a m.2 SSD (on PCI-E 8 Lanes) and a Soundcard (on the PCI-E 4 Lanes) means i use 28 Lanes.
But my CPU only support 16 Lanes means i lost some Performance; right??

My PC:

Corsair RM850 
Gigabyte GA-Z87X-D3H Motherboard
i7 4790K 

32GB RAM @ 2133MhZ
Quadro P4000
Kingston m.2 SSD

X-Fi Soundblaster (Creative)
 

gpu will run a 8x, m.2 at 4x, and the sound card at 4x. Make sure the gpu is the top card. The difference between a gpu at 16c and 8x is literally negligible.

[Manuel Bachmann]

ok, GPU 8X vs 16X for Image & Video editing....

 

[Bajantechnician]

1 minute ago, Bachmann Photography said:

ok, GPU 8X vs 16X for Image & Video editing....

 

the difference between 16x and 8x is essentially none.

[Manuel Bachmann]

ok thnx, so im fine with it is know. Thnx guys

[Bajantechnician]

1 minute ago, Bachmann Photography said:

ok thnx, so im fine with it is know. Thnx guys

Yeah, no prob. Good luck

[Admiral Breaker]

On 10/6/2015 at 6:17 AM, nunya bus said:

yeah the 20 lane limit for skylake is keeping me from getting it. im still thinking about a 5930K as the 40 lanes.... or I wait for broadwell E. I wonder if that will have DMI 3 or a new chipset or will that us X99 still? 

That is why I went with the 5930k. I just added a 256gb NVME m.2 and according to PCpartpicker the system cannot do SLI due to the NVMe. However my CPU has 40 lanes... either way I was going from a Sandy Bridge i5-2500 and jumped to the i7-5930k. 

[The Benjamins]

On 11/28/2017 at 7:19 AM, Stefan Payne said:

...until Treadripper comes along...

 

The CPU has 64 PCIe lanes. 

All of them. Even the rather cheap 1900X. 

Why would you want a Skylake-X anyway??

Only if you don't buy that POS, you can force Intel to change their politics...

Its actually only 60 usable, 4 is dedicated to the X399 chipset.

[Stefan Payne]

Just now, The Benjamins said:

Its actually only 60 usable, 4 is dedicated to the X399 chipset.

AH, OK...

 

But I wonder why nobody does a "Chipsetless" Board for AMD Ryzen or Threadripper. ITs entirely possible because those have everything integrated....

 

And that is still around +50% from Intel LGA2066 and 44 Lanes

[Ace McPlane]

On 10/6/2015 at 6:19 AM, Sakkura said:

Okay, so watching the 5820K vs 6700K showdown video on Vessel, I came across a common misconception about the PCIe lanes on Skylake and Haswell-E unfortunately being perpetuated by @Slick.

 

The Core i7-6700K, and all the other Skylake CPUs, offers 16 PCIe 3.0 lanes. The chipset offers additional PCIe lanes separately from that. For the Z170 chipset, that's 20 PCIe 3.0 lanes, a huge boost over Z97 which had just 8 PCIe 2.0 lanes. Luke was just saying 20 lanes, which is not really correct; there is a kernel of truth to it though, because while the chipset offers all those lanes, it's still only connected to the CPU by a DMI 3.0 link that's equivalent to 4 lanes of PCIe 3.0 (in addition to the 16 lanes directly from the CPU). Still, you can hook up lots of PCIe 3.0 SSDs to the chipset just fine without affecting lanes for the GPU(s). Just don't expect RAID0 to give you like 10GB/s combined bandwidth.

 

As for the Core i7-5820K, it offers the well-known 28 PCIe 3.0 lanes directly from the CPU. But again, the chipset offers additional PCIe lanes. In this case, X99 is far inferior to Z170, because it only offers 8 PCIe 2.0 lanes (just like Z97).

 

Here are the block diagrams showing what I explained above:

 

Hidden Content

 

Do note that the lower-end chipsets, eg. H110, cut down on the PCIe connectivity on offer.

Thanks for the info!

I've been avoiding putting too many cards in my system because of the 16 PCIe lane limit, but I guess I have room for a few more..

Hell, I could even do SLI while keeping my USB expansion card if 1080's weren't so damn expensive right now or if I didn't buy one that you can't possibly find anywhere now. The OC gaming edition.

I regret nothing.

 

[kingmustard]

Does this thread still need to be pinned?

[TheProfosist]

On 9/22/2018 at 7:16 AM, kingmustard said:

Does this thread still need to be pinned?

its still good info could be retitled though