IIS, SMB, FTP?

[johnt]

I have two PCs running Windows and I need to transfer large files between them, roughly 60 to 90 GB each, somewhere on the order of about half a terabyte per transfer. Both machines are on the same home network (this is not running in a public or office setting). I've been using SMB and the transfer speed has been very fickle. Both machines are configured on a 2.5 Gbps setup (NICs, switch). And SMB has only saturated my network bandwidth once in all my testing. I've tried to research some other methods and quickly found videos of FileZilla making full use of the tuber's bandwidth, but then I read forums where some folks are just anti-FTP. I also saw Windows has IIS that can be setup to do FTP without any extra software. But I'm skeptical if FTP through File Explorer is going saturate my bandwidth. I'm just not familiar enough with Linux to really move platforms. And FileZilla puts more work on me instead of the automated scripts I have now.

 

So what's the deal with FTP? Is it a problem over a home network? Is IIS faster than SMB? Is there something else I can look into?

[OddOod]

Are you reading and writing from SSDs? You might be bottlenecked by R/W speed
What speeds are you getting to with SMB? I've gotten 2.3ish Gb/s on SMB
What are the specs of the computers?

As for FTP, it's fine, just annoying to set up. 

[AbydosOne]

22 minutes ago, johnt said:

I've been using SMB and the transfer speed has been very fickle. Both machines are configured on a 2.5 Gbps setup (NICs, switch). And SMB has only saturated my network bandwidth once in all my testing.

7 minutes ago, OddOod said:

What speeds are you getting to with SMB? I've gotten 2.3ish Gb/s on SMB

I pretty routinely get 1GBps from SSD* to SSD* with SMB on 10GbE. In all likelihood, SMB and 2.5GbE aren't your bottlenecks here.

* = okay, technically they're both arrays of SSDs (ZFS mirror and Windows stripe, respectively).

[johnt]

4 minutes ago, OddOod said:

Are you reading and writing from SSDs? You might be bottlenecked by R/W speed
What speeds are you getting to with SMB? I've gotten 2.3ish Gb/s on SMB
What are the specs of the computers?

I've tried with NVME drives and spinners. Both seem to max out around 100 MB/s when they are rated for around 250 (or higher with NVME). I can usually get the spinners to 230 MB/s when connected to the same PC.

 

The computers are an i5 12400 and an i3 12100, 32 and 16 GB of memory. NICs are built in 2.5 Gbps, switch/network is Ubiquiti. Nothing else going on over the network (no updates, no streaming). Here is a screenshot of test I just put together from one NVME on the "host" to an NVME on the "client" over SMB (the spinners are capped around the same speed). Screenshots from two different points in the transfer:

 

[johnt]

18 minutes ago, AbydosOne said:

I pretty routinely get 1GBps from SSD* to SSD* with SMB on 10GbE. In all likelihood, SMB and 2.5GbE aren't your bottlenecks here.

I can see your point, but I have no idea what else to point the finger to? My network does not have any bandwidth caps. The devices are connected to the same switch through 2.5 Gbps, and the switch is connected through 10 Gbps DAC to the gateway/console. Both devices are using a short 3 feet ethernet cable. Both devices are on the same VLAN. The i3 machine was a fresh install from yesterday. The i5 was built a little while ago, but both have NIC drivers installed from ASUS.

 

Where else can I look?

[johnt]

Some improvement after restarting the machines:

 

[AbydosOne]

14 minutes ago, johnt said:

Some improvement after restarting the machines:

 

That looks suspiciously like 1GbE... What are your NICs currently linking at? (Network Status > View hardware and connection properties)

[johnt]

4 minutes ago, AbydosOne said:

That looks suspiciously like 1GbE... What are your NICs currently linking at? (Network Status > View hardware and connection properties)

Not sure why... Task Manager is reading 1.1 to 1.2 Gbps during transfers on both devices. The i3 is on the left and the i5 is on the right. The network console also reports they are connected at 2.5 GbE.

 

What is the maximum transmission unit??

 

 

 

[OddOod]

1 hour ago, johnt said:

computers are an i5 12400 and an i3 12100

Can you show us what the CPU utilizations are on these when transferring? 
Make sure to Change graph to > Logical processors 

[mariushm]

May want to try tools like FastCopy and see if they do better than Windows Explorer : https://fastcopy.jp/

 

You should really get a couple 10g or 25g network cards and connect your computer to the other using a DAC cable... then you'll get real speeds... and cards aren't that expensive, like $50 for a 25g card and maybe 40-50$ for a dac cable.

 

I like setting up Filezilla FTP server on one machine,  and Filezilla FTP client on the other machine.

 

On Filezilla FTP client, you only need to check the option to treat all files as binary transfers and you're good to go.

 

It can help by transferring multiple files in parallel - when you have lots of small files it can happen to not get high speeds as the connection keeps closing and opening and speeds need to ramp up after each connection starts

 

In Filezilla server you can also set it up to listen on multiple interfaces (ex if you have a network card with 2 10g ports), you can then open two server connections with filezilla ftp client (and transfer a folder using one interface, and another folder using the other interface, so potentially get up to 20gbps)

 

 

 

 

ps. there's also some FTP clients that support downloading with multiple threads from ftp , if my memory is correct older versions of Flashget

[johnt]

10 minutes ago, OddOod said:

Can you show us what the CPU utilizations are on these when transferring? 
Make sure to Change graph to > Logical processors 

 

CPU

 

Ethernet

[johnt]

I did a bandwidth test using my gaming PC as the server, which also has a 2.5 Gbps NIC. It's not breaking the 2 Gbps mark, but the file transfer speeds should still be higher than ~100 MBps.

 

 

[johnt]

21 minutes ago, mariushm said:

May want to try tools like FastCopy and see if they do better than Windows Explorer : https://fastcopy.jp/

 

You should really get a couple 10g or 25g network cards and connect your computer to the other using a DAC cable... then you'll get real speeds... and cards aren't that expensive, like $50 for a 25g card and maybe 40-50$ for a dac cable.

 

I like setting up Filezilla FTP server on one machine,  and Filezilla FTP client on the other machine.

 

On Filezilla FTP client, you only need to check the option to treat all files as binary transfers and you're good to go.

 

It can help by transferring multiple files in parallel - when you have lots of small files it can happen to not get high speeds as the connection keeps closing and opening and speeds need to ramp up after each connection starts

 

In Filezilla server you can also set it up to listen on multiple interfaces (ex if you have a network card with 2 10g ports), you can then open two server connections with filezilla ftp client (and transfer a folder using one interface, and another folder using the other interface, so potentially get up to 20gbps)

 

ps. there's also some FTP clients that support downloading with multiple threads from ftp , if my memory is correct older versions of Flashget

I'll have to give fastcopy and filezilla a try. I tested IIS for an FTP in Windows, and the speeds were even slower. Not sure what the heck is going on.

[OddOod]

16 minutes ago, johnt said:

CPU

dang. I was hoping this was the bottleneck. 
This is a shot in the dark, but maybe try disabling Remote Differential Compression on both machines?
https://www.easeus.com/pc-transfer/file-transfer-speed-slow-between-computers-on-lan.html#:~:text=10/8/7-,Solution 2. Turn Off "Remote Differential Compression",-The Remote Differential
It does need a restart after disabling

[johnt]

@OddOod @mariushm

 

I disabled Remote Differential Compression on both machines and it didn't make a difference. I am still between 80 to 120 MBps through SMB Direct, but it is very inconsistent. FTP through FileZilla was extremely stable at 135ish.

 

I wish I didn't know any better. But when I started this network adventure, I tested my LAN transfer speed between PCs and I was getting 280 MBps consistent for the entire transfer. So I know it's possible and I'm not just chasing a myth. I feel like it shouldn't be this hard. Thanks for your help guys.

 

What do you guys think about enabling legacy SMB through Windows instead of relying on SMBD alone? The other issue I've been reading about is having the devices on the same subnet. I realized I have each device on a separate VLAN, but I have rules to let them communicate, plus the subnets are open to each other in the firewall rules. I have them on different VLANs for different security reasons. It's a pain to change them, but do you guys think that would make a difference?

 

 

[AbydosOne]

2 minutes ago, johnt said:

I realized I have each device on a separate VLAN, but I have rules to let them communicate, plus the subnets are open to each other in the firewall rules. I have them on different VLANs for different security reasons. It's a pain to change them, but do you guys think that would make a difference?

Yes. The CPU in your router still needs to process each connection through each rule, even if the rule is "don't do anything". This will severely hamper connection speed.

[johnt]

4 minutes ago, AbydosOne said:

Yes. The CPU in your router still needs to process each connection through each rule, even if the rule is "don't do anything". This will severely hamper connection speed.

I was afraid someone would think it's a good idea lol

 

What about IPv6? I had to enable it to get the VPN to work over t-mobile. But I noticed it added a ton of rules also. Do you think disabling IPv6 will make a difference for LAN? 

[Lurick]

15 minutes ago, johnt said:

I was afraid someone would think it's a good idea lol

 

What about IPv6? I had to enable it to get the VPN to work over t-mobile. But I noticed it added a ton of rules also. Do you think disabling IPv6 will make a difference for LAN? 

No it won't make a difference. Intra-VLAN traffic is being processed by the router's CPU, that's the bottleneck.

[mariushm]

29 minutes ago, johnt said:

@OddOod @mariushm

 

I disabled Remote Differential Compression on both machines and it didn't make a difference. I am still between 80 to 120 MBps through SMB Direct, but it is very inconsistent. FTP through FileZilla was extremely stable at 135ish.

 

That's odd because that's just a bit more than 1 gbps. 

 

Could you maybe try connecting the two computers together through an ethernet cable for a quick test , without a router/switch ?   The cards should do auto pair detection and connect at 2.5gbps

 

Set the IPs manually if needed, for example one to 192.168.0.101  and the other to 192.168.0.102  and both 255.255.255.0 subnet mask and try a bunch of ftp transfers in parallel.

 

edit: btw ... just noticed in your last pictures that the connection is  1gbps ... note how under ethernet card name on both it says 1 gbps.

Is your card even connecting to the rest of your network at 2.5gbps ?

On 6/11/2024 at 9:24 PM, johnt said:

 

Ethernet

 

 

[johnt]

19 minutes ago, Lurick said:

No it won't make a difference. Intra-VLAN traffic is being processed by the router's CPU, that's the bottleneck.

I'm a bit shocked to see traffic rules would decrease transfer speeds by almost half. But it is totally worth a shot. I can live with these two machines on the same VLAN.

[Lurick]

21 minutes ago, mariushm said:

edit: btw ... just noticed in your last pictures that the connection is  1gbps ... note how under ethernet card name on both it says 1 gbps.

Is your card even connecting to the rest of your network at 2.5gbps ?

 

 

It's how Windows scales the output, not an indicator of negotiated speed. You need to hit or exceed the upper value for it to resize to the next value.

[johnt]

19 minutes ago, mariushm said:

That's odd because that's just a bit more than 1 gbps. 

 

Could you maybe try connecting the two computers together through an ethernet cable for a quick test , without a router/switch ?   The cards should do auto pair detection and connect at 2.5gbps

 

Set the IPs manually if needed, for example one to 192.168.0.101  and the other to 192.168.0.102  and both 255.255.255.0 subnet mask and try a bunch of ftp transfers in parallel.

 

edit: btw ... just noticed in your last pictures that the connection is  1gbps ... note how under ethernet card name on both it says 1 gbps.

Is your card even connecting to the rest of your network at 2.5gbps ?

I am going to try that next (direct connection). I suppose if that doesn't work, then I have a NIC issue. If it does work, I have a network issue. Joy!

 

That 1 gbps designation is the maximum of the chart. It scales as the usage gets higher or lower. I had a different screen shot showing both these devices are connected to the switch at 2.5 gbps. I can confirm it on the switch as well, plus the link speed light on each NIC is green, which means 2.5 gbps connection according to the manuals (they are essentially the same motherboards, anyway).

[johnt]

@Lurick @mariushm @AbydosOne @OddOod

 

I just moved the machines to the same VLAN and disabled the firewall rules that were no longer necessary to allow connections between them. The SMBD transfer speeds immediately bumped up. I was getting 270-278 MBps consistent (NVME to NVME) with my kids watching on Plex and wife on Teams and my work VPN on a different laptop. Rust to rust was 230 MBps consistent, which is basically the max speed of that spinner pair.

 

I'm super mixed on this result. First, I am very thankful it is resolved, but second, DANG that is a huge hit to transfer speeds because of separate VLANs and two firewall rules. Third, I cannot believe it's resolved and the weekend hasn't even started yet!!! I have over 3 TB of data backlogged to transfer.

 

Thank you guys for walking me through this one. Honestly this was stressing me out.

[OddOod]

On 6/14/2024 at 1:14 PM, johnt said:

huge hit to transfer speeds because of separate VLANs and two firewall rules

My guess is that the VLAN interlink or the firewall is only running at gigabit speeds. It's a bummer. What were you using hardware-wise for those?

[johnt]

15 minutes ago, OddOod said:

My guess is that the VLAN interlink or the firewall is only running at gigabit speeds. It's a bummer. What were you using hardware-wise for those?

UDM SE for gateway and general console management. This device is rated for 3.5 Gbps routing with IDS/IPS over WAN, so it was tough for me to think simple VLAN rules were causing a problem. The switch is a Unifi Enterprise 24-port POE model, connected to the UDM with a short SFP+ DAC. And both computers were connected to 2.5 Gbps ports on the switch with RJ-45 cables.