SMB PC-PC - 350MB/s with 5Gbps connection, just right or AMD bottlenecking?

[scottiecb19]

Hi everyone.

 

So I'm trying to get a 'simple' file transfer in place between my main PC and a Server PC. Both are running Windows 10, main PC has a 3900X, 32GB Ram, 1 x 1Gbps on board and a separate PCI-e with 2 x 2.5Gbps connections. Server PC has the same network setup, with a Ryzen 3600 and 32GB Ram.

 

I have the two PC's connected directly to each other via the two 2.5Gbps connections, each has it's own IP, and I read not to assign a gateway or DNS to this (contrary to Level1's advice on this, though I found it made no difference to transfer speeds). Both NIC's are RSS capable, and transferring a large .zip file or video file, I can see at both ends the throuughput is being shared equally between the two connections on each computer, so the SMB function must be working. However, with a total expected throughput of 5Gpbs, I was expected transfer speeds over 600MBps, but I'm only getting 300-350MBps for a few seconds, before it drops into the 200's and sometimes less. Looking on task manager, I can see both CPU's are maxing out on all cores, so I was wondering if the speed could be CPU bottlenecked (should add, there's nothing going through the 1Gbps connections, which are on a separate subnet). I'm also transferring from m.2 to m.2 and they are coping just fine in task manager. I wouldn't have thought so, watching the Quadruple your Network Speed Video from a few years back where they were using older processors, I would have thought a higher throughput would have been possible. I've tried everything I can think of and everything of mutliple sites and forums, and I still don't get close to the speeds I anticipated.

 

In anticipation of the 'why do you need that speed', simply put, I don't, but that's my latest little project so that's what I want to do.  

 

Any help or guidance would be appreciated!

[GeorgeB123]

If your CPUs are maxed out then that's the bottleneck, yes. I wonder, however, why would they be maxed out by a simple transfer over ethernet, even tho it's at a relatively high speed. I've seen even stronger CPUs than the 3900X max out when downloading games from steam at 10Gbps but that also includes decompression and installation of the data being transfered and it's at higher transfer speeds. Maybe the configuration could be optimized.

[mariushm]

Make a 16-24 GB ram drive on the computer and save to the RAM drive and repeat the test. I use ImDisk Toolkit but there's other freeware applications that can do ram drives.

At least this would take out of the equation the storage on one side.

You shouldn't get you cpu saturated with this... try and see the cpu usage with just one link instead of two - could be the cpu usage is caused by the combination of data from both ports in memory or some other stuff related to using two connections.

 

Also try using something smarter than Windows Explorer, like for example FastCopy - https://fastcopy.jp/  -  or TeraCopy or even Total Commander's copy (you have options to configure buffer sizes when reading files, which can speed up transfers)

[scottiecb19]

Thank you both for the replies.

 

1 hour ago, mariushm said:

Make a 16-24 GB ram drive on the computer and save to the RAM drive and repeat the test. I use ImDisk Toolkit but there's other freeware applications that can do ram drives.

 

Checking the disks being written from and to, both are barely going over 10% usage, so do you think the RAM disk is necessary. I put an SSD in my server to ask as a cache for transfers, but AMD's software is worse than temporamental and wouldn't install, so I'll use it as a manual cache if needed. 

1 hour ago, mariushm said:

Also try using something smarter than Windows Explorer, like for example FastCopy - https://fastcopy.jp/  -  or TeraCopy or even Total Commander's copy (you have options to configure buffer sizes when reading files, which can speed up transfers)

I also tried FastCopy and TeraCopy and both struggled with transfer speeds above 200MB/s. I tried adjusting settings, but neither would go higher. Interestingly, the CPU's were still maxed out on with these speeds. Not sure what is happening here, but something doesn't feel right.

[Alex Atkin UK]

Have you looked to see if your cards supports Interrupt Moderation?

 

I had to disable it on my 10Gbit cards to get performance up which is kinda funny as the whole point of it is to reduce CPU usage, but something just wasn't quite right with how it was functioning.

 

Interestingly performance as fine going Linux to Linux, the problem was going from the Linux NAS to Windows 10 and yes even Windows 10 to Windows 10 is impacted.  I've found there is something fundamentally broken in Windows 10 networking lately.

[scottiecb19]

3 hours ago, Alex Atkin UK said:

Have you looked to see if your cards supports Interrupt Moderation?

 

I had to disable it on my 10Gbit cards to get performance up which is kinda funny as the whole point of it is to reduce CPU usage, but something just wasn't quite right with how it was functioning.

 

Hi, thanks for the reply. So both cards support interrupt moderation, I disabled on all 4 connections, and still no luck - initial peak at 350MB/s then a drop to sub 200MB/s.

[scottiecb19]

Have tried a number of 'make your network faster' sites, most were a complete flop, unsurprisingly, not as a criticisim but I accept each issue is likely to be unique. I had a quick thought, as a test, to try an internal SSD-SSD transfer, and after an initial transfer speed peak of over 1GB/s, I dropped to a consistent 400-420MB/s - could this be a processor/SSD issue, not a hardware one?

[scottiecb19]

So quick update, after leaving the settings, I'm getting consistent 380MB/s for the first 2gb or so of transfer before it drops. Still not quite sure what to try next. Smb is clearly working, so where is the throttling occurring? 

[Blue4130]

32 minutes ago, scottiecb19 said:

So quick update, after leaving the settings, I'm getting consistent 380MB/s for the first 2gb or so of transfer before it drops. Still not quite sure what to try next. Smb is clearly working, so where is the throttling occurring? 

Could it be the drives themselves running out of cache? What happens when you do a file transfer between two drives on the same machine? Just a shot in the dark...

[heimdali]

What does iperf say?

 

I would say that if your SSDs can sustain writing about 200MB/sec indefinitely, you can't complain.

 

Perhaps try it with a single 2.5G connection so as not to overwhelm your CPU.

 

[scottiecb19]

17 hours ago, Blue4130 said:

Could it be the drives themselves running out of cache? What happens when you do a file transfer between two drives on the same machine? Just a shot in the dark...

Quote

On 9/19/2021 at 3:01 PM, scottiecb19 said:

Have tried a number of 'make your network faster' sites, most were a complete flop, unsurprisingly, not as a criticisim but I accept each issue is likely to be unique. I had a quick thought, as a test, to try an internal SSD-SSD transfer, and after an initial transfer speed peak of over 1GB/s, I dropped to a consistent 400-420MB/s - could this be a processor/SSD issue, not a hardware one?