M.2s, memory and PCIe lanes

[jonnyyyl]

Having been complacent about SSDs for a while I'm really struggling with about the latest m.2 technology and developments. And I figure that I might as well just ask for help, after becoming even more confused through reading more, most probably because I don't have the basics right.

 

Below is a summary of my peanut understanding, pls kindly comment on my understanding. Much appreciated:

 

1. M.2 is an interface, like SATA interface. Because of the current CPU architecture, most motherboards only support 1 or 2 slots. All mobo manufacturers will indicate which m.2 supports which technology/bandwidth speeds

 

2. NVMe and 3D NAND are technologies which the memory utilizes, as of now NVMe is the currently, as of date of this piece, fastest and up-to-date technology. NVMe/3D Nand can come in the form of SSDs or M.2. But because of the bottle neck of SATA3 ports(6gb/s) , m.2 (32gbs/s) is increasingly employed and the preferred form factor. 

 

3. Most motherboards, to compliment current CPU architectures, still support PCIe X16, X8, X4. No boards, at least premium/prosumer grade products like the ROG series, to my knowledge support 2 x PCI 3.0 x16 or more lanes. 

 

4. PCIe 3.0 x16 lane transfer speeds theoretically 16gb/s, x8 at 8gb/s etc. So: 1) running NVMe m.2 on PCIe 3.0 x8 or x4 lane will see a bottleneck, 2) if I have 2 PCIe NVMe SSDs in the x16 AND the x8 slots, the speeds, like SLI/Crossover, will be at x8 speeds only. 

 

5. A scenario to highlight the problems:

a) if one were to get a full system running on several NVMe m.2s , there is no such support for it. The maximum is running 2, one on the mobo, and the other in the PCIe 3.0 x16 lane.  

b) if one were to fill the PCIe 3.0 x16 lane with the NVMe SSDs, and IF one has, say GTX 1080Ti running at base at 11gb/s, putting that GPU in the x8 lane will see a bottleneck.

c) if one even were to try with getting a 10Gb/s network PCIe adaptor, only the x4 lane is available, and the adaptor will see a bottleneck. 

d) because ALL 3 lanes are used, so everything is running at x4 speeds?!

e) Mobo real estate is a very big issue, and no CPU architecture/Mobo can catch up with latest developments in memory speed.

 

Thank you for the patience for reading this and putting up with my potato knowledge, and I could really use the help to clear matters up. 

Thx!

[Daniel Z.]

2 minutes ago, jonnyyyl said:

Having been complacent about SSDs for a while I'm really struggling with about the latest m.2 technology and developments. And I figure that I might as well just ask for help, after becoming even more confused through reading more, most probably because I don't have the basics right.

 

Below is a summary of my peanut understanding, pls kindly comment on my understanding. Much appreciated:

 

1. M.2 is an interface, like SATA interface. Because of the current CPU architecture, most motherboards only support 1 or 2 slots. 

 

2. NVMe and 3D NAND are technologies which the memory utilizes, as of now NVMe is the currently, as of date of this piece, fastest and up-to-date technology. NVMe/3D Nand can come in the form of SSDs or M.2. But because of the bottle neck of SATA3 ports(6gb/s) , m.2 (32gbs/s) is increasingly employed and the preferred form factor. 

 

3. Most motherboards, to compliment current CPU architectures, still support PCIe X16, X8, X4. No boards, at least premium/prosumer grade products like the ROG series, to my knowledge support 2 x PCI 3.0 x16 or more lanes. 

 

4. PCIe 3.0 x16 lane transfer speeds theoretically 16gb/s, x8 at 8gb/s etc. So: 1) running NVMe m.2 on PCIe 3.0 x8 or x4 lane will see a bottleneck, 2) if I have 2 PCIe NVMe SSDs in the x16 AND the x8 slots, the speeds, like SLI/Crossover, will be at x8 speeds only. 

 

5. A scenario to highlight the problems:

a) if one were to get a full system running on several NVMe m.2s , there is no such support for it. The maximum is running 2, one on the mobo, and the other in the PCIe 3.0 x16 lane.  

b) if one were to fill the PCIe 3.0 x16 lane with the NVMe SSDs, and IF one has, say GTX 1080Ti running at base at 11gb/s, putting that GPU in the x8 lane will see a bottleneck.

c) if one even were to try with getting a 10Gb/s network PCIe adaptor, only the x4 lane is available, and the adaptor will see a bottleneck. 

d) because ALL 3 lanes are used, so everything is running at x4 speeds?!

e) Mobo real estate is a very big issue, and no CPU architecture/Mobo can catch up with latest developments in memory speed.

 

Thank you for the patience for reading this and putting up with my potato knowledge, and I could really use the help to clear matters up. 

Thx!

pretty good, 3d vnand is a TLC manufacturing technique. the bandwidth for 8x PCIE 3.0 is more than enough for a gpu today. 

[Sakkura]

1. No. M.2 is a slot. It's comparable to the SATA connector, but not the SATA interface. An M.2 slot can either provide a SATA connection or a PCIe connection (or in less common cases also some other stuff).

 

2. NVMe is a storage protocol, which is the sort of software layer on top of the electrical connection. In this case, an SSD on a PCIe connection can use the NVMe storage protocol, or the older AHCI protocol. Not sure what you mean by "SSDs or M.2" as an M.2 SSD is still, you know, an SSD. Maybe you mean 2.5" vs. M.2. 3D NAND is just the type of flash memory inside the SSD, obviously any kind of SSD can use it. Even USB flash drives can use it.

 

3. Should be some boards on the enthusiast platform that can do PCIe 3.0 x16/x16. Not that it really matters for the vast majority of uses.

 

4. Theoretical maximum is 15.75 GB/s to be exact. There's still a little bit of encoding overhead, even though PCIe 3.0 has moved from 8b/10b to 128b/132b encoding. Running an NVMe M.2 drive on PCIe 3.0 x4 will not see a bottleneck. I think you may be confusing bits and bytes here.

 

5. You can have more than 2 M.2 NVMe SSDs. But yes, it gets awkward for a variety of reasons.

[Leonard]

24 minutes ago, Daniel Z. said:

pretty good, 3d vnand is a TLC manufacturing technique.

what?

 

TLC =  Triple-level cell there is also SLC = Single-level cell and MLC = Multi-level cell.

 

[agent_x007]

Ad. 1.
M.2 utilises PCI-e OR SATA connection routes (depending on "key" used by slot : B-Key = SATA, M-Key = PCI-e).

Ad. 2. 
From memory technology, you can have NAND, 3D NAND or Optane Memory on M.2 drive. NANDs usually are MLC or TLC (depending on price tag and drive).
NVMe and AHCI are used as protocols to communicate with OS directly [NVMe], or with drive controller [AHCI].
M.2 B-Key can only utilise SATA/AHCI
M.2 M-Key can do PCI-e with AHCI or NVMe (depending on drive you use)

Ad. 3. LGA 1366 and LGA 2011(-3) are your friends 

Ad. 4. There are no M.2 type drives with x8 or x16 lanes of any PCIe speed.
Max. is x4. and that gets us to 4GB/s max. (by math and that's with PCI-e 3.0 CPU/MB).
In practice - ~3,5GB/s Read speeds can be reached.

Ad. 5. Riser cards and PCI-e adapters (to M.2) are your friends  
But they are limited to speed of slot they are connected to.
If you need a lot of PCI-e 3.0 lanes, buy Ryzen or LGA 2011 platform.
Here's a test of NVM-e PCI-e 3.0 x4 drive, on PCI-e 2.0 x4 slot (PCI-e to M.2 adapter was used) : LINK.

PS. PCI-e sharing scheme is MB's thing.
Some may not even have a second PCI-e 3.0 x16 while others share lanes from SATA Express and M.2.
Sharing works only in one way :
You can't have two devices using PCI-e x4 speed, when the PCI-e lanes available to both of them are "x4" max.
They will be slowed down to x2 speed by default (or one will be disabled for the other to use full x4 speed available).
Because of that, they shoudn't slow each other down when working at the same time.
It's similar thing to using NVMe PCIe 3.0 x4 drive in PCIe 2.0 x4 slot.
Drive will work at full speeds as long as you don't exeed 2GB/s transfer limit on operations you are doing.

[jonnyyyl]

7 hours ago, Sakkura said:

4. ......Running an NVMe M.2 drive on PCIe 3.0 x4 will not see a bottleneck. I think you may be confusing bits and bytes here.

 

 

Ah. Could you be so kind to elaborate?

Cheers

[Sakkura]

36 minutes ago, jonnyyyl said:

Ah. Could you be so kind to elaborate?

Cheers

A PCIe 3.0 x4 connection has a bitrate of 32 Gb/s, which is equivalent to 4 GB/s. Due to encoding overhead, the maximum usable throughput is 3.94 GB/s.

 

The fastest M.2 NVMe SSDs can only transfer at around 2.7 GB/s.

[jonnyyyl]

Ah,

Cool thanks.

 

One last question

If the current transfer limits are 4GB/s, and fitting them in the PCIEx4 lane is not a problem, then I can technically put on 2-4 nvme on adaptor cards and place it in the x16 lane. Would that work? What are the bottlenecks, if any? 

[Sakkura]

In theory, yes. But the PCIe lanes directly from the CPU normally won't support booting from a storage drive, and on a mainstream motherboard you would be giving up the ability to have a graphics card.