M.2 NVMe SSD uses wrong transfer mode?

[ygrabo]

Hi!
It appears that my second onboard M.2 NVME SSD is running at PCIe Gen 3.0 x1 (or PCIe 2.0 x2) speeds instead of PCIE 3.0 x2 or x4.
I determined this based on CrystalDiskMark 8 where my first SSD manages ~2.5GB/s R/W and the second one only manages 0.8GB/s R/W. This actually happens with 2 different PCIe 3.0 x4 SSDs.
Any suggestions on what might happen?

Background info:

Motherboard:
Gigabyte Aorus B550 Elite V2 (see attached files)

- The first M.2 SSD slot (M2A_CPU) can achieve up to PCIe 4.0 x4 through the CPU (and it does)

- The second M.2 SSD slot (M2B_SB) can achieve up to PCIe 3.0 x4 through the chipset (doesn't seem like it does)

The only hardware connected to the MB are the Graphics card (RTX2060) and the 2 M.2 NVMEs
Link to manual

SSDs:

- System: Samsung 970 EVO 500GB (PCIe 3.0 x4)

- Data: Sabrent Rocket 2TB (PCIe 3.0 x4)

- Data: ADATA XPG SX8100 4TB (PCIe 3.0 x4)
Crystal Diskinfo reports PCIe 3.0 x4 for all drives, but is that what the drive reports it can do, or what the actual connection is (seems wrong)

(Waiting for a PCIe slot adapter to install the third drive permanently, but both "data" drives have the same speed in that second M.2 slot. The speed the third drive will run at in the slot adapter is not the question)

CPU:
Ryzen 5 5600X (Allows for PCIe 4.0)

PCIe speeds:
PCIe 3.0 x1 maxes out at ~1GB/s
PCIe 3.0 x4 maxes out at ~4GB/s

BIOS Settings:
The only related settings I could find in the BIOS didn't help (I tried more or less all settings), as they seem to relate to the CPU-bound lanes only.

- PCIEX16 Bifurcation: Allows you to determine how the bandwidth of the PCIEX16 slot is divided. Options: Auto, PCIE 2x8, PCIE 1x8/2x4, PCIE 2x4/1x8.

- PCIe Slot Configuration: Allows you to set the operation mode of the PCI Express slots to Gen 1, Gen 2, Gen 3, or Gen 4

 

Any help would be greatly appreciated. Thanks!

 

 

 

 

 

 

 

[HanZie82]

The chipset has 4 lanes and needs to share that with many components.
 

[Frizz]

If the GPU is using 16 lanes, then I think I found your answer.

GPU + 3 drives that can use 4 pcie lanes.

The 5600x can only do 24 lanes.

IIRC your chipset only uses pcie lanes from your CPU.

16+4+4+4 = 28 lanes. Your drive can't use all the lanes it wants

[Alex Atkin UK]

8 minutes ago, Frizz said:

If the GPU is using 16 lanes, then I think I found your answer.

GPU + 3 drives that can use 4 pcie lanes.

The 5600x can only do 24 lanes.

IIRC your chipset only uses pcie lanes from your CPU.

16+4+4+4 = 28 lanes. Your drive can't use all the lanes it wants

That's not usually how it works as the chipset will have a MUX that splits those chipset lanes, otherwise it would be impossible to have three M.2 slots, USB, SATA, sound, PCIe x1 and x4 sockets, etc - as they all share those chipset lanes.  They're all contending for the same bandwidth but should have all their lanes functioning, thus when testing an SSD it should perform properly as the rest of those things wont be using much bandwidth.

 

Although I do see other motherboards mention that the B550 chipset only exposes PCIe 3.0x2 to M.2 drives.

 

This is where the big difference between B550 and X570 comes in, the latter runs the chipset at PCIe 4 and MUXes at PCIe x4 with more lanes. https://www.electronicshub.org/b550-vs-x570/

[ygrabo]

19 minutes ago, Alex Atkin UK said:

That's not usually how it works as the chipset will have a MUX that splits those chipset lanes, otherwise it would be impossible to have three M.2 slots, USB, SATA, sound, PCIe x1 and x4 sockets, etc - as they all share those chipset lanes.  They're all contending for the same bandwidth but should have all their lanes functioning, thus when testing an SSD it should perform properly as the rest of those things wont be using much bandwidth.

 

Although I do see other motherboards mention that the B550 chipset only exposes PCIe 3.0x2 to M.2 drives.

I would agree with that. It would mean that you could not even use the basic features of the motherboard... All Ryzen 5000 CPUs including the R9s have "only" 24 lanes.

x2 speed would be ok though. TBH, 800MB/s is plenty, but if something is just misconfigured, might as well fix it.

[Alex Atkin UK]

Just now, ygrabo said:

I would agree with that. It would mean that you could not even use the basic features of the motherboard...
x2 speed would be ok though. TBH, 800MB/s is plenty, but if something is just misconfigured, might as well fix it.

I seem to recall it was the reason I went X570, although I had forgotten the specifics until I looked it up just now.

[ygrabo]

18 minutes ago, Frizz said:

If the GPU is using 16 lanes, then I think I found your answer.

GPU + 3 drives that can use 4 pcie lanes.

The 5600x can only do 24 lanes.

IIRC your chipset only uses pcie lanes from your CPU.

16+4+4+4 = 28 lanes. Your drive can't use all the lanes it wants

it is 24 PCIe 4.0 lanes.
4 lanes are used to interconnect with the chipset (which would be 8GB/s).

Found this from guru3d:
Ryzen 5000 CPUs will have a total of 24 PCIe Gen4 lanes. Four out of the twenty-four are used for the interconnect to the motherboard chipset, leaving 20 lanes Gen 4.0 for other utilization. 16 lanes (PCIe x16) are intended for graphics cards that are connected as x16 or two at x8. Seen from Gen 3.0 one a Gen 4.0 x8 link would offer similar to PCIe 3.0 x16 bandwidth. Four more PCIe lanes from the CPU are designed for fast storage such as PCIe 4.0 NVMe compatible SSDs. In addition to all that, a sidenote - AMD has integrated USB 3.2 Gen2 support into the CPU.

All this seems to be available only with an X570 chipset. more on that further below.

[ygrabo]

2 minutes ago, Alex Atkin UK said:

I seem to recall it was the reason I went X570, although I had forgotten the specifics until I looked it up just now.

Right...I found this table. So would that mean that everything else shares 10 x4 lanes... suuuuurely one of them could go to my poor SSD
 

 

[Frizz]

30 minutes ago, Alex Atkin UK said:

That's not usually how it works as the chipset will have a MUX that splits those chipset lanes, otherwise it would be impossible to have three M.2 slots, USB, SATA, sound, PCIe x1 and x4 sockets, etc - as they all share those chipset lanes.  They're all contending for the same bandwidth but should have all their lanes functioning, thus when testing an SSD it should perform properly as the rest of those things wont be using much bandwidth.

 

Although I do see other motherboards mention that the B550 chipset only exposes PCIe 3.0x2 to M.2 drives.

 

This is where the big difference between B550 and X570 comes in, the latter runs the chipset at PCIe 4 and MUXes at PCIe x4 with more lanes. https://www.electronicshub.org/b550-vs-x570/

That was what I said? I didn't say it was impossible to use the lanes, just that the drive is contending with other devices for those lanes. Hence it limits its ability to "stretch its legs".

 

But to be clear when I said the chipset shares lanes from your CPU, I just meant it didn't have built in pci e lanes. Not that the chipset literally just took every lane a device requested haha, imagine that

[Alex Atkin UK]

12 minutes ago, Frizz said:

That was what I said? I didn't say it was impossible to use the lanes, just that the drive is contending with other devices for those lanes. Hence it limits its ability to "stretch its legs".

 

But to be clear when I said the chipset shares lanes from your CPU, I just meant it didn't have built in pci e lanes. Not that the chipset literally just took every lane a device requested haha, imagine that

It was this that was misleading:

Quote

Your drive can't use all the lanes it wants

It CAN use all the lanes it wants, it just wont necessarily get all the bandwidth it wants from those lanes.

 

59 minutes ago, ygrabo said:

As the Transfer Mode is listed twice I suspect that states both the current mode and the drives maximum mode, so it IS using PCIe 3.0 x4.  Why its performing so slow I do not know.

Have you tried simply copying a large file between the two drives to confirm the real-world performance?

[Frizz]

12 minutes ago, Alex Atkin UK said:

It was this that was misleading:

It CAN use all the lanes it wants, it just wont necessarily get all the bandwidth it wants from those lanes.

 

 

So curious, as I may be missing something, how is it that with his 16X GPU, and 3 4X drives that it could use all the lanes it wants. The OP did say "

Samsung 970 ... (PCIe 3.0 x4)

- Data: Sabrent ... (PCIe 3.0 x4)

- Data: ADATA ... (PCIe 3.0 x4)"

I could totally be missing something, but 16 + 4 + 4 + 4 is greater then the total PCIe lanes the CPU has, are you saying that some lanes are basically being used by more then one device concurrently? If so that is something new I learned!

[ygrabo]

2 minutes ago, Frizz said:

So curious, as I may be missing something, how is it that with his 16X GPU, and 3 4X drives that it could use all the lanes it wants. The OP did say "

Samsung 970 ... (PCIe 3.0 x4)

- Data: Sabrent ... (PCIe 3.0 x4)

- Data: ADATA ... (PCIe 3.0 x4)"

I could totally be missing something, but 16 + 4 + 4 + 4 is greater then the total PCIe lanes the CPU has, are you saying that some lanes are basically being used by more then one device concurrently? If so that is something new I learned!

the indicated speeds are the manufacturer specs of the drives.

currently, only one of the two data drives is connected at any given time. I only have 2 M.2 slots (sorry if that was unclear, I will clarify).

[Blue4130]

17 minutes ago, Frizz said:

So curious, as I may be missing something, how is it that with his 16X GPU, and 3 4X drives that it could use all the lanes it wants. The OP did say "

Samsung 970 ... (PCIe 3.0 x4)

- Data: Sabrent ... (PCIe 3.0 x4)

- Data: ADATA ... (PCIe 3.0 x4)"

I could totally be missing something, but 16 + 4 + 4 + 4 is greater then the total PCIe lanes the CPU has, are you saying that some lanes are basically being used by more then one device concurrently? If so that is something new I learned!

It isn't 16+4+4+4, it is 16+4+4 with the last 4 going through the chipset's 4 lanes.

16 for gpu

4 for first m.2

4 for chipset (this 4 includes secondary m.2,ancillary pcie slots, usb, networking, audio, everythingelse) - These all share the bandwidth of the 4 lanes. And that could be the reason for the slow speeds. If you have lots of traffic on the NIC or USB or other slots, the m.2 will need to slow down and share bandwidth.

[ygrabo]

46 minutes ago, Alex Atkin UK said:

Have you tried simply copying a large file between the two drives to confirm the real-world performance?

I did.
I created a 21GB file on the slow drive and copied it from the slow drive to the fast drive (D: to C:). It started at about 400MB/s and maxed out at about 700MB/s before slowing down again to 600MB/s.
I then renamed the file and copied it back to the slow drive. This is where caching might be an issue (for the test)? It starts with ~2GB/s, but then quickly slows down to about 800MB/s (hinting to some form of 4-6GB cache).
That's weird because it can only push 2GB/s if the other side accepts it, cache or not. But that drive *should* be able to be faster in sequential reads. I think that was actually the case on my old motherboard (X370 chipset and PCIe slot adapter).

The test was repeated twice.
The files are called the same because they stem from repeats.
 

[ygrabo]

*deleted reply*

[ygrabo]

16 minutes ago, Blue4130 said:

It isn't 16+4+4+4, it is 16+4+4 with the last 4 going through the chipset's 4 lanes.

16 for gpu

4 for first m.2

4 for chipset (this 4 includes secondary m.2,ancillary pcie slots, usb, networking, audio, everythingelse) - These all share the bandwidth of the 4 lanes. And that could be the reason for the slow speeds. If you have lots of traffic on the NIC or USB or other slots, the m.2 will need to slow down and share bandwidth.

According to some specs posted above, it would be 10 lanes going to the chipset.
Anyway, the other PCIe slots are empty, USB didn't do anything except drive mouse and keyboard, my audiointerface is external via USB and that is low throughput, no SATA drives, networking was chilling...
I will do tests tomorrow unplugging everything, but I don't have high hopes.

[Blue4130]

1 hour ago, ygrabo said:

According to some specs posted above, it would be 10 lanes going to the chipset.
Anyway, the other PCIe slots are empty, USB didn't do anything except drive mouse and keyboard, my audiointerface is external via USB and that is low throughput, no SATA drives, networking was chilling...
I will do tests tomorrow unplugging everything, but I don't have high hopes.

No, it is 4 lanes GOING to the chipset. The chipset has ten lanes worth of devices that need to share 4 lanes of communication between the cpu and chipset.

[ygrabo]

Thanks everybody so far. I should still get more speed I reckon, even with what all of you said (with the up and down lanes). I will get the M.2 to PCIe slot adapter today - maybe that will help diagnose what is going on.

[Alex Atkin UK]

4 hours ago, ygrabo said:

Thanks everybody so far. I should still get more speed I reckon, even with what all of you said (with the up and down lanes). I will get the M.2 to PCIe slot adapter today - maybe that will help diagnose what is going on.

Would sure be bizarre if that made a difference, seeing as it should be wired to the chipset just the same.

[ygrabo]

Right, so I ran each drive individually in the M.2 or in the PCIe adapter and they run at ~850MB/s each. I also ran both at the same time, and then we get 900MB/s shared between the two.
So basically, the motherboards come with 2x M2 slots, but the second one can never run at its intended speed as soon as you use the network or something? What the hell? Misleading advertisement?

 

[ygrabo]

I deactivated the SATA controller (no drives connected anyway) and the onboard ethernet, that didn't change a thing.
I also changed PCIe "bifurcation" to 4x4 and other options - nothing changed.

Are you telling me that the motherboard advertises a second M.2. slot that should run at PCIe 3.0 x4 speeds but it never will, even with a Ryzen 9 ?
What about all the x4 slots that only run in 1x mode "according" to the drives ... surely something is off o.O

[ygrabo]

UPDATE
Gigabyte tech support has confirmed that the drives SHOULD run at the speeds I expected, i.e. x4 in the second M.2 slot (via chipset) and x2 in the PCIe slot with M.2. to slot adapter.
They therefore presume something is broken and asked me to return or RMA it.
I just installed an X570 board... everything is speedy.

[Ashutoshsharma]

I am also facing this issue and asked the question in this forum