Slow file transfer via SMB over VPN

[damnfinecoffee]

Hi guys,

 

This one has our network installer, ISP and hardware manufacturer stumped so if any of you can figure this out.. heroes!

 

At our office we have a 12-bay Synology NAS which is sat on our gigabit ethernet lan. We also have a Draytek 3910 firewall which handles our VPN and a 900 Mbps down / 500 Mbps up fibre internet connection.

I have a large (20+GB) file on the NAS which I can move over the LAN to my PC Workstation at around 112MB/s.

 

I now have my workstation at home, and I am connected to the office via VPN. My home internet connection is 420 / 50 (same ISP as the office).

 

If I navigate to the web UI of the synology via our public IP I can grab this same 20GB file and it will download at around 50MB/s. No issues here.

If I then connect to our VPN and navigate to the web UI of the synology via it's local IP I can grab this same 20GB file and pull it down at around 42.6MB/s. I guess this is still about right as there is a little overhead for the VPN.

 

But our issue is this:

 

When accessing the share using file explorer in Windows 10 (still via VPN and local IP) the transfer speed appears to max out at around 25MB/s.

What's even more bizarre is if I change our VPN protocol from SSL to L2PT then this drops even further to around 10MB/s.

But transferring the file via the web interface from the same local ip still hits 40+ MB/s.

 

Does anyone have any idea what's going on here?

I can provide further info if needed

 

Thanks in advance!

[Kilrah]

Probably whatever you're using as the VPN server having a hard time with encrypting the SMB packets...?

[damnfinecoffee]

6 minutes ago, Kilrah said:

Probably whatever you're using as the VPN server having a hard time with encrypting the SMB packets...?

Hi Kilrah,

 

Thanks for the info, is this a common issue? Can you advise of a route to rectify this?

 

We're using the Draytek Vigor 3910 firewall router which handles our VPN and DHCP.

Draytek themselves insist there shouldnt be an issue, and switching VPN protocols shouldnt make any difference to our transfer speeds but clearly it does, and they didnt mention SMB packets at all..

 

Thanks

[damnfinecoffee]

So after some further testing I have discovered the following:

 

These are all whilst connected to our VPN via SSL protocol:

 

Download via web gui via public IP (VPN bypassed) = 50MB/s

Download via web gui via local IP = 41MB/s

Download via File Explorer (Win 10) via local IP = 25MB/s

 

I appreciate that there will be some overhead with the VPN, which is why I'm seeing 50 drop to 41, but does anyone know why this takes a massive hit when using File Explorer. As @Kilrah said this could be due to the packet type, but this seems lke a huge difference, or am I just expecting too much here and this is to be expected?

 

Thanks in advance

[charlie_root]

The issue is that SMB is a block based protocol whereas HTTP is a streaming protocol. SMB 1.0 was really bad about this as it could only read 64k at a time, then it would have to contact the server and ask for the next 64k, etc. This chattiness results in a lot of overhead, and the VPN would have to encrypt and decrypt each packet. 

 

SMB 2.0 addressed these problems to some extent, but it is still not nearly as efficient as HTTP. You may want to see if you can determine whether this transfer is using SMB 1.0 or 2.0, as it may be falling back to 1.0. In either case though, it will never be as good as a streaming protocol like HTTP or FTP/SFTP. 

 

See this blog post from Microsoft about the problem( for 1.0; it's from 2004): https://docs.microsoft.com/en-us/archive/blogs/neilcar/smbcifs-performance-over-wan-links

[damnfinecoffee]

Thanks @jsf that's really interesting.

 

I was kind of hoping this was the answer but this makes it even more of a head scratcher.

The NAS I'm pulling the test file from is SMB2 (in the office I can move this file locally at about 112MB/s).

Also, to confuse things even further, we do also have an older NAS which is SMB1. I enabled SMB1 support on my Win10 remote workstation and used this same test file to downloaded via the same method over the VPN. Based on your info I expected this to be even slower, but it actually downloads faster. Not massively, but definitely consistently faster. My results are as follows:

 

NAS A (SMB2): Download via SSL VPN using File Explorer: 25MB/s

NAS B (SMB1): Download via SSL VPN using File Explorer: 29MB/s

 

It also looks like that 25-29MB/s limit gets shared into to two when I download from both NAS's simultaneously.

If I download from one NAS via the VPN and also download a second file from it simultaneously but via the public ip (VPN bypassed) then my full bandwidth gets shared up.

 

I've attached examples to hopefully better explain..

 

Thanks again

 

 

[charlie_root]

2 hours ago, damnfinecoffee said:

Thanks @jsf that's really interesting.

 

I was kind of hoping this was the answer but this makes it even more of a head scratcher.

The NAS I'm pulling the test file from is SMB2 (in the office I can move this file locally at about 112MB/s).

Also, to confuse things even further, we do also have an older NAS which is SMB1. I enabled SMB1 support on my Win10 remote workstation and used this same test file to downloaded via the same method over the VPN. Based on your info I expected this to be even slower, but it actually downloads faster. Not massively, but definitely consistently faster. My results are as follows:

 

NAS A (SMB2): Download via SSL VPN using File Explorer: 25MB/s

NAS B (SMB1): Download via SSL VPN using File Explorer: 29MB/s

 

It also looks like that 25-29MB/s limit gets shared into to two when I download from both NAS's simultaneously.

If I download from one NAS via the VPN and also download a second file from it simultaneously but via the public ip (VPN bypassed) then my full bandwidth gets shared up.

 

I've attached examples to hopefully better explain..

 

Thanks again

 

 

That's interesting. There's a lot of factors involved, and the protocol version is just one of them.

 

You may be able to increase the TCP window size to get better performance here. Basically the TCP window is the amount of data sent before the receiver has to acknowledge. When the TCP window limit is reached, the sender will pause and wait for an ACK before continuing to transmit, or even retransmitting the data it just sent. 

 

Increasing this window size is a strategy used in Long Fat Networks (LFN) where the bandwidth is wide, but the latency is high (satellite links to other countries are a classic example). The high latency means the receiver's ACK will be slow to get there, so reducing the number of ACKs that need to be sent can generally improve performance since the sender will transmit more data before pausing and waiting for the ACK.

 

With that said, you want to be careful here; not all firewalls can handle large TCP windows. It's also no panacea. Most operating systems have TCP autoscaling turned on by default, which will automatically adjust the TCP window so that manual configuration is not necessary. I'm also not convinced that this would solve the problem, since SMB is a really inefficient protocol for a high latency link. It's something to try, but this issue would require more investigation before going this route. 

 

 

[Bad5ector]

Have you guys stumbled upon this:

 

https://www.mirazon.com/issues-with-smb-file-transfer-performance-over-vpn/#:~:text=If you did Google for,%2C they're all there.&text=Reduce the MTU size on,this to 1400 or 1350.

[damnfinecoffee]

Thanks guys, that explains a lot but also confuses things further from my tests this morning. I ran the following with no other traffic on our network:

 

I setup a second Workstation at this location, same configuration as my primary Workstation.

 

Workstation A: Connect to VPN SSL - Download file via File Explorer = 25MB/s

Workstation B: Connect to VPN SSL (different account) - Download file via File Explorer = 25MB/s

 

If I run these at the exact same time, I can see that the 25MB/s gets split between the two.

Workstation A = 15MB/s

Workstation B = 10MB/s

 

To confuse things further, I removed Workstation B from my network and connected to a 4G network via my phones hotspot.

I then connected to the VPN and started downloading the same file, this hits 5MB/s which is about right for my cellular connection.

But.. Workstation A remained at 25MB/s whilst this process was happening simultaneously.. the second workstation had no impact on this apparant 25MB/s limit whilst it was also downloading..

 

Does this make any sense to you guys?

In my head this suggests the the limit is based on location/IP as opposed to different VPN accounts.

 

Thanks again

[damnfinecoffee]

Hi Guys,

 

Another update, just to confuse things EVEN FURTHER...

 

I have just discovered that if I connect to our VPN via IKE v2 in OSX then there is absolutely no issue at all, my download hit almost 50MB/s immediately!

Great! The only problem is.. I simply cannot connect this way in Win 10. I've tried via both the Windows VPN setup and also the Draytek Smart VPN Client, both methods appear to reach the server and ask for username/password, but as soon as I enter them they both produce these errors:

 

IKE authentication credentials are unacceptable

 

I'm using the exact same login details as I used in OSX, and that connects instantly!

 

Any ideas?

 

Thanks