slow network transfer speeds

[emosun]

Trying to move files from one machine to another via the network and am only getting about 10MBps. The drive on the other machine internally can transfer files at least at sustained 40MBps and the network even if it was only gigabit should be able to handle more than 10. Already tried turn off "Remote Differential Compression" and Disable "Large Send Offload (LSO)".

 

It's like 6 terabytes of files so getting it up to 30 or 40 would cut the transfer down to a day or so instead of a week

[Needfuldoer]

10 MB/sec sounds like you're only getting a 100 megabit connection.

 

What NIC does that machine have? Does another system get a full gigabit connection with that same patch cable?

[emosun]

5 minutes ago, Needfuldoer said:

10 MB/sec sounds like you're only getting a 100 megabit connection.

 

What NIC does that machine have? Does another system get a full gigabit connection with that same patch cable?

I'm not 100% sure as i just fumble and google my way through networking but here's the two boards im using with the networking info in the specs

https://www.supermicro.com/products/motherboard/archive/QPI/5500/X8DTH-iF.cfm
https://www.asus.com/us/Motherboards-Components/Motherboards/Workstation/Pro-WS-WRX80E-SAGE-SE-WIFI/

[mariushm]

Install a FTP server on one machine (I use Filezilla FTP server, it takes less than 5 minutes to install and set up a basic user account) and use a FTP client (Filezilla FTP client is fine) on the other machine to transfer files. 

By default Filezilla FTP client does up to 10 simultaneous connections, so if you're transferring lots of small files it works much faster.

 

Just don't forget to set in Filezilla FTP client's options to treat all transfers as binary, otherwise some files may be changed  (set default to Binary and UNCHECK "Treat files without extension as ASCII files and  Treat dotfiles as ASCII files)  

 

 

 

This is assuming the hardware bit (the cables, the network cards) are all fine.  With one transfer at a time, it's slow because with each transfer connection is closed, new connection is opened, and it takes a second or so for the transfer speed to ramp up, so with lots of small files, it takes a lot of time to close and open connections, and with small file sizes it rarely ramps up to full speed.

With multiple simultaneous connections, you know something always transfers. 

 

Another option would be to just zip / tar some folders with lots of small files (use 7-zip and choose no compression or ultra fast compression), and compress a folder, you can even choose to save the zip directly on that other computer - it's often faster because that single connection is kept open and the speed can ramp up to maximum possible.

 

[mariushm]

7 minutes ago, emosun said:

I'm not 100% sure as i just fumble and google my way through networking but here's the two boards im using with the networking info in the specs

https://www.supermicro.com/products/motherboard/archive/QPI/5500/X8DTH-iF.cfm
https://www.asus.com/us/Motherboards-Components/Motherboards/Workstation/Pro-WS-WRX80E-SAGE-SE-WIFI/

Shame you don't have a 10g network card in that supermicro server. 

Make sure your data is sent through the ethernet cable and not the wireless on that Asus motherboard. Disable Wifi if you're not sure. 

 

If you connect to a switch with the 10g card on your asus board, you could have the 2 1gbps connections connected to the switch and transfer data through each 1g connection, for a maximum of 2 gbps... 

[emosun]

18 minutes ago, Needfuldoer said:

10 MB/sec sounds like you're only getting a 100 megabit connection.

4 minutes ago, mariushm said:

 

ok turns out im an idiot , the switch they are connected to is a 100mb switch. lol. so yeah that would explain the issue. didn't even realize that till i checked just now

 

[Needfuldoer]

9 minutes ago, emosun said:

ok turns out im an idiot , the switch they are connected to is a 100mb switch. lol. so yeah that would explain the issue. didn't even realize that till i checked just now

 

Been there, done that!

 

If this was more than a one-off, I'd recommend picking up a couple 10 gig NICs and an SFP+ DAC cable.

[mariushm]

7 minutes ago, emosun said:

ok turns out im an idiot , the switch they are connected to is a 100mb switch. lol. so yeah that would explain the issue. didn't even realize that till i checked just now

 

If you don't have a gigabit switch, you should be able to connect the computers directly with a regular cable, but you'll have to manually assign the IPs and subnet mask to those ports in that case,  for example set 192.168.0.11 and 192.168.0.12   and subnet mask 255.255.255.0 

The network cards should detect a direct connection with a regular ethernet cable and automatically arrange the pairs of wires correctly and get a working network between the two network cards.

If that doesn't work, you can make a proper 1 gbps crossover cable by rearranging the order of wires in one of the connectors:

 

[emosun]

13 minutes ago, mariushm said:

you should be able to connect the computers directly with a regular cable

wow really? every fiber of my pc knowledge tells me that not supposed to work but thats pretty interesting lol

[mariushm]

36 minutes ago, emosun said:

wow really? every fiber of my pc knowledge tells me that not supposed to work but thats pretty interesting lol

Yes, really. See https://en.wikipedia.org/wiki/Medium-dependent_interface#Auto_MDI-X

Auto Pair Detection is a feature implemented in 1 gbps network cards and higher. Majority of network cards support it, but SOME are a bit buggy and don't do it right. That's why I offered the alternative of making a proper 1 gbps (or higher) crossover cable (the picture)

 

100 mbps (and older) network cards did not have auto pair detection, so using a crossover cable was a must. But 100 mbps crossover cables are even easier to make because 100 mbps cards only use 2 pairs of wires (the green and orange, pins 1,2,3 and 6 ... the other two pairs can be left untwisted as they're not used.) - that's why if you search Google for "crossover cable" you'll find lots of pictures where the blue and brown pairs are not twistes, those are 100 mbps maximum crossover cables.