Why is launching executables from a network location so slow?

[bcredeur97]

While normally this hasn't bothered me too much, I see it as a need for where I work as well. It's convienient to have programs stashed away on a network drive, that we can just quickly map and copy files from the drive to the desktop of the user's pc to install the software they need.

but that's the thing -- we always have to copy the file. We can't simply launch the executable on the network drive itself or it will take a day and a half to install a piece of software... yet we can just copy it over like lightning (literally 80-90MBps over wired gigabit usually) and launching it locally is quick....

 

I've noticed this issue with my NAS devices at home as well. data transfer is nice and fast, but if you launch an executable, network usage in resource monitor in windows will hover around 10mbps until the whole program is loaded -- almost as if it's an artificial limit.

 

Why is this the case? Is there anything that can be done to speed this up?

[KenjiUmino]

i noticed this as well - no clue what exactly causes it - but it's one reason why storing the steam library on the NAS to access it from multiple computers is a dumb idea.

[KarathKasun]

Most programs make calls to the filesystem libraries for disk access, this has to be re-encapsulated first to operate on the network.

 

Its likely that the network overhead for this type of access is much higher in Windows.  I don't notice this with NFS on Linux.  In fact, I can run a remote root with full speed.  The same thing can be done with Windows, but the network abstraction happens outside of the OS removing this problem.

[factorialandha]

Windows File shares are very chatty protocols, meaning latency is a pain in some cases,

 

Launching executables from a network location means it has to be chattier, depending on the application you launch

 

It also depends on what you are launching, and the latency between yourself and the network location, for example, i can launch a 250mb installation from a network share in the same amount of time i can do it locally.

 

However if you have a lot of shared resources stored in the network location too, there will be alot of back and forth over the network to load each resource when its requested, meaning latency wont be your friend at all.

 

You might also look at what version of SMB you are using, SMB 3 reduces the chattiness, maybe look at whether your devices support it, and if they do, force it to that protocol (this means windows xp machines i believe wont be able to access those resources) 

 

[bcredeur97]

21 minutes ago, KarathKasun said:

Most programs make calls to the filesystem libraries for disk access, this has to be re-encapsulated first to operate on the network.

 

Its likely that the network overhead for this type of access is much higher in Windows.  I don't notice this with NFS on Linux.  In fact, I can run a remote root with full speed.  The same thing can be done with Windows, but the network abstraction happens outside of the OS removing this problem.

hmm

12 minutes ago, factorialandha said:

Windows File shares are very chatty protocols, meaning latency is a pain in some cases,

 

Launching executables from a network location means it has to be chattier, depending on the application you launch

 

It also depends on what you are launching, and the latency between yourself and the network location, for example, i can launch a 250mb installation from a network share in the same amount of time i can do it locally.

 

However if you have a lot of shared resources stored in the network location too, there will be alot of back and forth over the network to load each resource when its requested, meaning latency wont be your friend at all.

 

You might also look at what version of SMB you are using, SMB 3 reduces the chattiness, maybe look at whether your devices support it, and if they do, force it to that protocol (this means windows xp machines i believe wont be able to access those resources) 

 

Well im not sure which SMB im using at home -- I assume 3 because i believe that is default in Win10 with the latest updates...

 

I know for sure at work we are using SMBv3 and we even have Jumbo packets enabled on our switches -- though I'm not sure if the clients utilize those??

[factorialandha]

8 minutes ago, bcredeur97 said:

hmm

Well im not sure which SMB im using at home -- I assume 3 because i believe that is default in Win10 with the latest updates...

 

I know for sure at work we are using SMBv3 and we even have Jumbo packets enabled on our switches -- though I'm not sure if the clients utilize those??

most clients probably arent jumbo frames aware, but if you are using 3 and it still slow, might be worth checking disk activity when waiting for the app to launch, see if its throwing requests out to different locations. Unfortunately latency issues around stuff like that are difficult to diagnosed (ive spent hours trying to diagnose some issues related to this)

 

Im assuming they are direct Client -> Server and you arent using any load balance / DFS set up?

[bcredeur97]

49 minutes ago, factorialandha said:

most clients probably arent jumbo frames aware, but if you are using 3 and it still slow, might be worth checking disk activity when waiting for the app to launch, see if its throwing requests out to different locations. Unfortunately latency issues around stuff like that are difficult to diagnosed (ive spent hours trying to diagnose some issues related to this)

 

Im assuming they are direct Client -> Server and you arent using any load balance / DFS set up?

Ahhh it seems the issue with jumbo frames is it’s not enabled on our server... and if we did we’d have to enable it on every single device. Not doable... not easily anyway.

 

so I created a network share on my work desktop and dropped an installer in it. And enabled jumbo frames...

 

then I went to another desktop and enabled jumbo frames, connected to the share, and boom. Basically a local install! 

[Lurick]

1 minute ago, bcredeur97 said:

Ahhh it seems the issue with jumbo frames is it’s not enabled on our server... and if we did we’d have to enable it on every single device. Not doable... not easily anyway.

 

so I created a network share on my work desktop and dropped an installer in it. And enabled jumbo frames...

 

then I went to another desktop and enabled jumbo frames, connected to the share, and boom. Basically a local install! 

You shouldn't have to enable Jumbo Frames on everything. At the start of an IPv4 stream from the server to another, lower MTU client, it will send a B back to the jumbo frame enabled box when it hits a smaller MTU interface saying "Packet too big" and it will drop the MTU down and resend. IPv6 does this with a different mechanism known as Path MTU Discovery.

[bcredeur97]

Just now, Lurick said:

You shouldn't have to enable Jumbo Frames on everything. At the start of an IPv4 stream from the server to another, lower MTU client, it will send a B back to the jumbo frame enabled box when it hits a smaller MTU interface saying "Packet too big" and it will drop the MTU down and resend. IPv6 does this with a different mechanism known as Path MTU Discovery.

How fast is this done though? Will it be a noticeable amount of latency on local clients that don’t have the large MTU size?

[Lurick]

4 minutes ago, bcredeur97 said:

How fast is this done though? Will it be a noticeable amount of latency on local clients that don’t have the large MTU size?

It should just be the first packet in a stream and should be pretty much negligible in terms of additional latency. You can double check though and connect two clients and set one to Jumbo and leave the other at 1500 and double check the latency to see how big the impact is.

 

The one thing I did forget about in my earlier post is that the SVI the client and server live on, or any L3 interface in between, will need to be enabled for jumbo frames and jumbomtu (or it's equivalent for the switch model) will need to be enabled globally on the switch (if the option isn't there it might be on by default) but that should be it.

[bcredeur97]

1 hour ago, Lurick said:

It should just be the first packet in a stream and should be pretty much negligible in terms of additional latency. You can double check though and connect two clients and set one to Jumbo and leave the other at 1500 and double check the latency to see how big the impact is.

 

The one thing I did forget about in my earlier post is that the SVI the client and server live on, or any L3 interface in between, will need to be enabled for jumbo frames and jumbomtu (or it's equivalent for the switch model) will need to be enabled globally on the switch (if the option isn't there it might be on by default) but that should be it.

On our particular avaya switch, it’s actually enabled by default and you can’t turn it off haha

 

That was the first thing I looked up. 

 

But yeah it does seem if you if you want to run executables from a network share, you need jumbo frames to do so. 

 

Installing a ~850MB application (well that was the installer size, actual install is probably a little bigger) 

took 4 mins 16 seconds directly from local hard drive

 

and took 5 mins 23secs from jumbo-frame enabled network share

 

but it took nearly an hour for our non-jumbo existing network share which is actually on a server with 10k rpm drives and tons of available bandwidth. Trouble is we’ll have to enable jumbo there... for that I have to talk to my supervisor about it tomorrow. 

 

But cool nonetheless

[factorialandha]

13 hours ago, bcredeur97 said:

On our particular avaya switch, it’s actually enabled by default and you can’t turn it off haha

 

That was the first thing I looked up. 

 

But yeah it does seem if you if you want to run executables from a network share, you need jumbo frames to do so. 

 

Installing a ~850MB application (well that was the installer size, actual install is probably a little bigger) 

took 4 mins 16 seconds directly from local hard drive

 

and took 5 mins 23secs from jumbo-frame enabled network share

 

but it took nearly an hour for our non-jumbo existing network share which is actually on a server with 10k rpm drives and tons of available bandwidth. Trouble is we’ll have to enable jumbo there... for that I have to talk to my supervisor about it tomorrow. 

 

But cool nonetheless

just out of interest, have you tried it using switch ports not configured using jumbo frames, we dont use jumbo frames here in the office im in, jumbo frames we only use for our core structure in the DC not in the actual offices, and we dont have such a large drop in speed when running things from the network (at least not here using the local file server and my machine both connected via GB non jumbo frames.

 

just be nice to see if it is the MTU check thats causing an odd heavy delay.

 

on a different note to someone elses comment "it should just be the first packet" i believe SMB is very chatty, so there might be a lot of "first packets" but i have not really checked that in depth and SMB V3 does fix a lot of the chattiness

[bcredeur97]

19 minutes ago, factorialandha said:

just out of interest, have you tried it using switch ports not configured using jumbo frames, we dont use jumbo frames here in the office im in, jumbo frames we only use for our core structure in the DC not in the actual offices, and we dont have such a large drop in speed when running things from the network (at least not here using the local file server and my machine both connected via GB non jumbo frames.

 

just be nice to see if it is the MTU check thats causing an odd heavy delay.

 

on a different note to someone elses comment "it should just be the first packet" i believe SMB is very chatty, so there might be a lot of "first packets" but i have not really checked that in depth and SMB V3 does fix a lot of the chattiness

I was running short on time yesterday, but this morning when I got in I went back on that PC and tried the install without jumbo frames enabled... it was still fast. Not quite as fast -- but still fast.

So this means something isn't right on our server. back to square 1.

[Lurick]

1 hour ago, factorialandha said:

on a different note to someone elses comment "it should just be the first packet" i believe SMB is very chatty, so there might be a lot of "first packets" but i have not really checked that in depth and SMB V3 does fix a lot of the chattiness

That was probably me. When I said that for the jumbo frame piece it would just be the first packet in the stream that's over the size of the MTU limit. Any extra chatter no related to or separate from that data stream shouldn't have to deal with it, so the box should be able to adjust for the rest of the stream when launching a program locally but the next program or relaunch will go through the same process of returning to "too big" response packet and backing off the packet sizes and resending.

 

If they packets aren't jumbo to begin with and/or part of a different stream then it won't have to back off and resend.