Slow transfer speeds over 40GBe + NVME

[Packet7hrower]

Hey Guys!

So quick rundown of the hardware:

• Client

  • Mellanox 40GBe Card - established 40GB link straight to unRAID

  • NVME SSD, tested @ 3.8GB Read, 3.1GB Write via Crystal Disk Mark

  • Windows 11

• unRAID

  • Mellanox 40GBe Card - established 40GB link straight to Windows 11

  • NVME Cache Drive - same drive that is on the client. Tested via unRAID (dd write command, 1.9GB/s)

  • unRAID 6.9.2

When I transfer a file over to unRAID from Windows 11, I'm only seeing around 7.2Gbps copy speeds via explorer/taskmanager. 

 

Since I know my client disk can read around 3.8GB, and unRAID Cache is writing around 1.9GB, I should be at least seeing around 15Gbps in Windows right?

 

Is this a limitation of Windows Pro? SMB? Are there settings I'm missing???

[Kilrah]

What's your CPU usage? SMB as a single thread process tends to become CPU-bottlenecked at those speeds...

[jj9987]

32 minutes ago, Kilrah said:

What's your CPU usage? SMB as a single thread process tends to become CPU-bottlenecked at those speeds...

Probably this.

 

But OP, check your units as well. That network card is 40 Gbps, that is 40 gigabits per second, which is ~5 gigabytes (GB) per second (1 byte (big B) = 8 bits (small b)). Windows shows transfer speed in bytes.

[StDragon]

46 minutes ago, Kilrah said:

What's your CPU usage? SMB as a single thread process tends to become CPU-bottlenecked at those speeds...

SMB has very little CPU cycle overhead when transferring just one file (say an ISO, or very large multi GB sized media file) regardless of the fact it would be a single threaded operation.

Now if you're transferring thousands of smaller files via Explorer.exe, then it's because of a single-threaded limitation due to all of the IOPS involved. In that scenario, using ROBOCOPY with the /MT:32 switch (/MT default is just 8 threads) would saturated the SMB link properly

[Packet7hrower]

So - CPU on client is a 8700k, CPU on unRAID is a 6700K.

 

30-40% total CPU on unRAID, max of 60% on a single core.

 

10% total CPU Usage on Client, maybe 20% on single core.

 

When xfering with Robocopy MT:32, results are about the same - around 7ish Gbps.

 

My test file, is a single .7z of 14GB of drone footage.

[Packet7hrower]

Update - I installed iperf3 on my unRAID Server, and on my Client OS.

 

Results avg 18Gbit/s, which is about 2250MBps.

 

Issue 1: I would of expected to see around 35Gbit/s over iperf.

 

Issue 2: I now know 100% that my Link can achieve 2,200MBps, and I know that my Cache SSD on unRAID can write 1,800MBps - this still brings me back to my original problem of - why are my transfers not at least hitting the ~1,800MBps speed of my drive?