Storage / File Transfer Issue

[LT1_Sam]

Hello all,

 

At my place of work, we deal with larger file sizes from AutoCAD. Typically anywhere between 3-5+ gigs. There are two main locations we work from, with a pretty decent 50/50 fiber connection shared between the two, we run into problems when transferring files from one office to the next. It ends up capping itself to around 350 kb/s, which is very frustrating and extremely time consuming. Our storage box (which is acting kind of like a server for our computers to upload/download files too) is losing connection randomly (I believe to be due to a hardware issue) and we need a longer term solution. Unfortunately, the budget isn't particular high so a Storinator solution isn't viable. We have about 10 TB of storage, which would be expanded upon in the future. Our normal file server for the rest of the company is roughly 2TB and there are no plans due to expense to upgrade that one. This has to be a separate solution. 

 

I've had a couple of ideas which might help, however I'm unsure which path to take and would appreciate any input/suggestions. 

1.) Build two storage desktops that would act like one server. Bridge them together so when one file is upload, it would be mirrored in the other office location to be used by those employees. Problem: 350 kb/s transfer speed between the two offices. 

2.) Get a used server and load it up with storage and put into our server racks. Again problem: 350 kb/s transfer speed.

3.) Cloud storage. Problem: Most bottleneck upload/download speed, regulates cost in time

 

Thanks,

Sam

[Nerdtality]

It seems to me you are probably sending data to the Cloud at a faster rate than it is able to write it to disk. Thanks to write cache/buffers it is still able to receive it at the faster rate (the peaks of the graph), but you can't keep receiving data without committing it to disk.

Eventually the buffer will run full and has to be written to disk. Meanwhile, the Cloud cannot receive data as fast as before, as it has no where to store it (buffer is full, and disks are slower). This is where you get the valleys of the graph.

It seems Windows is smoothing the throughput graph. With more precise graphs (say from Performance Monitor), you can actually estimate the write buffer size, by analysing the intervals and transferred bytes.

[LT1_Sam]

Would you say that limiting the amount of information that goes into buffer (stopping it from writing it to the disk) help alleviate this problem? 

[Catsrules]

Sounds like you really just need to figure out the 350 kb/s transfer speed problem. I am assuming the transfer speed is fine when your local to the servers correct? if this is the case it is your connection not the file servers. (although something is wrong with the one that keeps loosing connection. But that sounds like a different issue. We can ignore for the moment.

 

 

First off you need to verify your actually getting 50 mpbs upload and download at both locations. Do a speed test at both sites, use a few speed test providers to get a good result.  

Assuming your getting around what you should be, how are the two offices connected to each other? Guessing over a site to site VPN? If so do you know what type of VPN are you using? What are the routers you are using?

 

Did this problem just start happening or has it always been super slow?

Try sending a test file between the two offices using reep io see what speed your getting.

https://www.reep.io/

 

[LT1_Sam]

Yes, transfer speed locally is perfectly fine. It's not even terrible when at the other office and accessing smaller files. - The storage box keeps loosing connection which yeah, is a different issue to be address (but is still part of the larger picture of whats going on). 

 

We're using a Meraki VPN to talk to the main office, which we've had good results with before. I don't directly work with the large files that the autocad people do, but they've been telling me that this has been going on for about a year now. 

 

I'll work on testing the network connections and try using reep io to see what we're actually getting.