Large Server backup why are trasnfers slow.

[Bmoney]

Ubuntu server to Ubuntu server. 
ZFS flle system to ZFS file system 

Why is it so slow?

Average about 40MB/s of a 10Gb/s network connection?

 

Is there tuning I can do. Better app?
(RSYNC fuck a directory I can't get back not confident with it)

It bounces up to 200MB/s why can't I maintain that? 
No issue from Windows to Ubuntu. Getting 400MB/s to 750MB/s

Literally makes no sense. Please tell me I am stupid and there is a compression or encryption thing to turn off. 

[Windows7ge]

Speed is often dependent on the protocol. RSYNC is an application but what network protocol are you using? SAMBA(SMB/CIFS), NFS, SSH/SFTP?

[Bmoney]

Just now, Windows7ge said:

Speed is often dependent on the protocol. RSYNC is an application but what network protocol are you using? SAMBA(SMB/CIFS), NFS, SSH/SFTP?

Samba as my LAN is windows dependent. 

[Windows7ge]

1 minute ago, Bmoney said:

Samba as my LAN is windows dependent. 

Have you ran an iperf test between the two servers?

[Bmoney]

1 minute ago, Windows7ge said:

Have you ran an iperf test between the two servers?

Burning 1GB/s+

[Windows7ge]

3 minutes ago, Bmoney said:

Burning 1GB/s+

Have you tried copying a group of files without RSYNC?

[Bmoney]

3 minutes ago, Windows7ge said:

Have you tried copying a group of files without RSYNC?

Yes and speeds of 40MB/s is yes why I am here. 

 

[Windows7ge]

2 minutes ago, Bmoney said:

Yes and speeds of 40MB/s is yes why I am here. 

 

Just trying to eliminate what could be causing the problem.

 

What happens if you copy files in reverse? 2nd Ubuntu Server to 1st Ubuntu Server?

[Bmoney]

1 minute ago, Windows7ge said:

Just trying to eliminate what could be causing the problem.

 

What happens if you copy files in reverse? 2nd Ubuntu Server to 1st Ubuntu Server?

Same speeds
Turn off all necessary services
 

[Windows7ge]

2 minutes ago, Bmoney said:

Same speeds
Turn off all necessary services
 

So it's ~40MB/s in both directions.

1. It's not a network protocol issues
2. It's not an application issue
3. It's not a network configuration issue
4. It's not a storage speed issue

That's about the extent of my knowledge here. Any deeper issue in how the two servers communicate I can't really troubleshoot. Sorry.

[Bmoney]

1 minute ago, Windows7ge said:

So it's ~40MB/s in both directions.

1. It's not a network protocol issues
2. It's not an application issue
3. It's not a network configuration issue
4. It's not a storage speed issue

That's about the extent of my knowledge here. Any deeper issue in how the two servers communicate I can't really troubleshoot. Sorry.

Same here thank brutha

[Electronics Wizardy]

If its zfs to zfs, why not use zfs send? That way you don't have the overhead of a transfer protocol

 

What does top look like?

 

Is sync on ?

 

 

[beersykins]

Is it a bunch of small files?  That will kill your transfer rate regardless of protocol.

[Alex Atkin UK]

16 hours ago, beersykins said:

Is it a bunch of small files?  That will kill your transfer rate regardless of protocol.

Indeed, even NFS to NFS I find IO is dramatically bad going SSD to SSD even compared to SSD to HDD on the same PC.

 

I'd love to improve it but not found anything.

[beersykins]

3 hours ago, Alex Atkin UK said:

Indeed, even NFS to NFS I find IO is dramatically bad going SSD to SSD even compared to SSD to HDD on the same PC.

 

I'd love to improve it but not found anything.

Do you notice any difference doing something like rsync with the -z compression flag?  

[Alex Atkin UK]

14 hours ago, beersykins said:

Do you notice any difference doing something like rsync with the -z compression flag?  

Not sure, but it did make me curious and I tried using scp to transfer a large file and hit 260.9MB/s which seems pretty darn good for 2.5Gbit.

 

Anyway I don't want to steal focus from the person who started the thread, I only mentioned it as I suspect we may be hitting a similar bottleneck when it comes to the latency penalty for lots of smaller files.

[Bmoney]

What after some more research the problem seems to be that it take sync lots of time to start transfer end the process and starting the next. 

[Alex Atkin UK]

5 hours ago, Bmoney said:

What after some more research the problem seems to be that it take sync lots of time to start transfer end the process and starting the next. 

Do you mean each file or starting after issuing the rsync command?