PCI-SIG releases PCIe 6.0 Spec v1.0

[Lurick]

Summary

Today we have the next generation of PCIe, PCIe Gen6 or PCIe 6.0. This new generation will be able to handle 64GT/s which means 256GB/s on a PCIe Gen6 x16 link, double that of PCIe Gen5, or quadruple what today’s PCIe Gen4 servers can achieve.

 

Quotes

Quote

2022 will be the year where we see broad platform support for PCIe Gen5 so the next-generation Gen6 will not be in 2022. Still, having the standard with new Flits (fixed size flow control units) for lightweight FEC, PAM4 signaling, and more is a big deal. This means that the ecosystem can start building and validating designs to the new standard.

 

My thoughts

I'm really glad to see PCIe bandwidth moving forward at a rapid pace again after being stuck on 3.0 for what seems like far too long. Sure this won't benefit consumers in the short term but it could have a whole host of implications for data centers and "the cloud" in the coming years especially when it relates to super high speed host networking (think dual 400 or 800GbE NICs) and storage services.

 

Sources

https://www.servethehome.com/pci-sig-releases-256gbps-pcie-6-0-x16-spec/

https://pcisig.com/pci-express-6.0-specification

 

[porina]

A possible user scenario might be allowing more devices to hang off the CPU. Right now, the number of lanes can be a problem if you want lots of bandwidth to different devices. For example, if you wanted to run multiple PCIe SSDs, you'd typically have x4 off the CPU, and anything on top of that has to go through chipset unless you steal it from GPU lanes. PCIe 4.0 x4 could be replaced by 6.0 x1. Of course, this is based on some assumptions: that most use cases wont need to go past 8GB/s for a looooong time, and it'll take quite some time for 6.0 devices to exist. Personally I've found some limitation of the move from SATA to NVMe is that it is typically difficult to put more than a couple in one mid range system without stealing lanes from elsewhere.

 

Also consider, people push for more until we get to some good enough level. Once that is reached, it then becomes more a cost optimisation. Narrower but faster could enable that.

[williamcll]

maybe with speeds like these one day I will get 1Gbps on mobile data.

[Dabombinable]

1 hour ago, porina said:

A possible user scenario might be allowing more devices to hang off the CPU. Right now, the number of lanes can be a problem if you want lots of bandwidth to different devices. For example, if you wanted to run multiple PCIe SSDs, you'd typically have x4 off the CPU, and anything on top of that has to go through chipset unless you steal it from GPU lanes. PCIe 4.0 x4 could be replaced by 6.0 x1. Of course, this is based on some assumptions: that most use cases wont need to go past 8GB/s for a looooong time, and it'll take quite some time for 6.0 devices to exist. Personally I've found some limitation of the move from SATA to NVMe is that it is typically difficult to put more than a couple in one mid range system without stealing lanes from elsewhere.

 

Also consider, people push for more until we get to some good enough level. Once that is reached, it then becomes more a cost optimisation. Narrower but faster could enable that.

gen 6 x4 CPU-PCH interconnect would be pretty damn nice.

[Ydfhlx]

Sooo, new Radeon 7500XT with PCIe Gen 6 x1?

[Doobeedoo]

Evolving at good pace. It will take time before it comes in consumer platforms, though neat, a ton of IO and speed.

[PeachGr]

I m just lost. So we have different pcie for nvme and different for GPUz? I have them all gen 3 on an "old" B450

[tim0901]

2 hours ago, PeachGr said:

I m just lost. So we have different pcie for nvme and different for GPUz? I have them all gen 3 on an "old" B450

PCIe lane speeds are defined by four things:

 

1. The CPU

2. The motherboard

3. The PCIe device (eg GPU, NVMe storage)

4. Your BIOS settings.

 

Each component in the chain needs to support a PCIe generation to be able to run at those speeds. A Zen 3 CPU (which offers Gen 4 lanes) in a B450 motherboard (which only supports gen 3 speeds) will only support devices at gen 3 speeds. Similarly, a gen 3 NVMe drive won't run at gen 4 speeds if you put it in an X570 motherboard with a Zen 3 CPU. A 3080, however, could run at gen 4 speeds in the same system as a gen 3 NVMe drive - not all PCIe devices in the system have to run at the same speeds, but any trio of CPU, motherboard and PCIe device must support the same generation. If you wish you can choose to run devices at a slower speed by tweaking your BIOS settings, which can sometimes be useful for stability, but will of course reduce performance.

 

Up until now being able to tell which speed a PCIe lane can run at on a particular piece of hardware has been fairly straightforward. On B450 yes, all PCIe devices (are able to) run at gen 3 speeds. Zen 3 is similar. Each CPU offers 24 PCIe lanes - all of which run at (up to) Gen 4 speeds.

 

20 of these lanes go to your physical PCIe slots, while 4 go to your chipset, where they branch out to your USB ports, SATA ports and - potentially - a number of m.2 slots for NVMe storage, the exact number of which depending on the motherboard. These m.2 slots will run at gen 4 speeds, but given the uplink to the CPU is only 4 lanes, you may encounter a bottleneck if you try to run multiple NVMe drives through your chipset. As such, knowing which of your NVMe slots is connected directly to your CPU, rather than through your chipset, is important for optimal performance.

 

Some motherboards - on any platform - will have more physical PCIe slots than the CPU has PCIe lanes, at which point lanes are usually split up (eg a pair of GPUs may run at x8 each instead of a single GPU at x16) which (in most cases) would have the same effect on performance as running the effected devices at gen 3 speeds. This isn't that noticable for GPUs, but is important for storage devices, just like with the chipset.

 

Alder Lake is different though, in that it offers two sets of PCIe lanes that run at different speeds. Alder Lake CPUS have 16 PCIe gen 5 lanes that are available through your standard PCIe slots - either as a single 16x slot or as a pair of 8x slots. This is what you would use for your GPU or pretty much any add-in card. With current GPUs these lanes will run at gen 4 speeds and similarly you could choose to run them at gen 4 speeds in the bios. In addition to this, you get four PCIe gen 4 lanes dedicated for use by PCIe based storage - these are probably connected to an m.2 slot on most motherboards.

 

(In addition to this you get either 4 or 8 DMI 4.0 lanes depending on the chipset you chose, which go to the chipset and - just like with AMD's implementation - are then used for USB, SATA and further m.2 slots at PCIe gen 4 speed. This is basically identical to AMD's implementation, but uses a proprietary link and offers much more bandwidth to the chipset - DMI lanes have the same data rate as a PCIe lane of the same generation so with 8x DMI lanes you get twice the bandwidth to the chipset as you would from the 4x PCIe 4.0 lanes on a Zen 3 platform.)

 

In theory - therefore - a PCIe gen 5 NVMe drive could run at gen 5 speeds if plugged into the physical PCIe slot on an Alder lake motherboard (using an adaptor like this one), but would only run at gen 4 speeds if plugged into any of the m.2 slots.

 

Does it make a difference today? No. There are no PCIe gen 5 GPUs on the market, nor are there any commercially-available PCIe gen 5 storage devices for consumers. The main benefit is that 16 5.0 lanes is equivalent to 32 4.0 lanes in terms of bandwidth, meaning Alder Lake is very appealing for those who wish to connect large amounts of PCIe devices to their PCs.

 

It is likely that we will get PCIe gen 5 GPUs in the near future - I would be surprised if Nvidia and AMD didn't support it with their next gen GPUs - but I wouldn't expect to see much of an uplift in terms of performance, especially given how little PCIe gen 4 matters here. Gen 5 NVMe drives are just around the corner and to some people will be very important, but given that a SATA SSD is "good enough" for most people already, with gen 4 NVMe drives being considered overkill for most consumer PCs, I can't imagine them seeing mainstream adoption anytime soon. As such, while it may seem silly, Alder Lake's PCIe lane distribution shouldn't be much of an issue for the "future proofness" of the system.

[williamcll]

7 hours ago, PeachGr said:

I m just lost. So we have different pcie for nvme and different for GPUz? I have them all gen 3 on an "old" B450

you should read the manual for your motherboard, it will tell you where your CPU's PCIs are allocated.

In most cases There should be just enough PCI lanes for at least one NVME SSD and one PCI3x16 device such as a GPU.

[DoctorNick]

Yes maybe we'll see this in 2025