10gig Bottleneck

[TheTwist]

I am unable to get 10 gig speed connection between Synology NAS and Workstations.

 

Attached is a rough diagram of a network setup and spec of each machines.

 

I tried to copy a 7GB folder and it doesn’t copy as quickly as I think it should (55.2 MB/s). 

 

Troubleshoots I have done: 

• Updated Nic on the Workstations
• Checked packet loss between PC and Switch and then Synology and Switch. 
• Unplugged and tested copying through each individual Cat6 cable between the switch and server
• Replaced network cables
• Checked the Synology logs for anything weird 
• Tried a file copy using FTP
• Swapped out both PCI 10 gig NIC cards
• Done another test with another Workstation:
  • PC1 – Synology   Same problem (55.2 MB/s)
  • PC2 – Synology  Same problem (48.5 MB/s)
  • PC1 – PC 2 works like a dream (377 MB/s)

It seems like it is aimed at the Synology but what I don’t know what It used to work before. Any ideas?

[Senzelian]

From what I can see my initial prediction would be, that your RAID array simply can't write data that fast.

What is the exact storage configuration on the NAS and what is it exactly that you tried to copy?

[TheTwist]

So with my test I copied a 7GB folder with small files inside. 

We also create large files up to 20GB from the NAS takes quite some time to open. 

 

What do you you suggest to do with the recreating the RAID array to get these speeds.  We are currently running  8 4TB drives using RAID 6 ext4   

[BloodKnight7]

What speeds are the drives? Are all of those mechanical? are they 5400rpm? if so... you should be getting 60-80MB/s per drive depending on the operation....granted RAID 6 has to calculate a dual parity so its way slower to write against other options like RAID 10, but its faster for reading purposes.

 

Are you using jumbo frames? flow control? 

[TheTwist]

@BloodKnight7 

 

Yeah they are all mechanical WD Red SATA III - 6Gb/s, 7200rpm.

 

I did have a go at using jumbo frames but didn't make a difference.

[BloodKnight7]

7 hours ago, TheTwist said:

@BloodKnight7 

 

Yeah they are all mechanical WD Red SATA III - 6Gb/s, 7200rpm.

 

I did have a go at using jumbo frames but didn't make a difference.

So... you should be getting then 80MB/s-100MB/s per drive... try disabling flow control from your network cards. the 10Gbps switch you are using...I guess its non-managed? or if its managed have you configured jumbo frames in those ports? if the answer is no then thats why jumbo frames didnt did any difference, you need to enable it everywhere. (sorry I dont know anything about netgear swtiches)

[TheTwist]

14 hours ago, BloodKnight7 said:

So... you should be getting then 80MB/s-100MB/s per drive... try disabling flow control from your network cards. the 10Gbps switch you are using...I guess its non-managed? or if its managed have you configured jumbo frames in those ports? if the answer is no then thats why jumbo frames didnt did any difference, you need to enable it everywhere. (sorry I dont know anything about netgear swtiches)

 

Yeah it is managed switch. Got Jumbo frames  set up on both Workstation, Switch and Server. 

 

I guess that its the  Drives in the NAS which is making it a slower speed compared to the work stations.  What would you recommend to speed the drives up so the copy is above 100MB/s

[igormp]

I believe it'd be a good idea if you could do a network only test with something like iperf, just to make sure that the actual network is ok and that the problem lies somewhere else:

 

You can add it through the following docker image: https://hub.docker.com/r/networkstatic/iperf3

And run it with:  sudo docker run -it --rm -p 5201:5201 networkstatic/iperf3 -s

And in a client you can run: iperf -c YourNasIP

[TheTwist]

Here are the results

[igormp]

Huh, looks like you're barely getting over Gbit instead of 10Gbit, so it seems that there may be a problem somewhere in your network.

 

Just to be sure, you could try some parallel calls from the client side with: iperf3 -P 10 -c YourNasIP

[TheTwist]

Wonder what it is then that is causing it. 

 

Here is that iperf3 -P 10 -c YourNasIP command attached

 

 

iperf10.txt

[igormp]

Still not quite 10gbit (7~8gbit), buy way better than regular gbit. This should net you something close to 1GB/s in raw transfer speeds, maybe you're being limited by disks as other people mentioned.

[TheTwist]

Hmm just ran Iperf between 2 work stations on the same network  link. And that is  giving 3-4gbit...  would that indicate something to do with the switch? Even thought it says 10G Full under Physical Status    

 

Even though copy speed is a lot faster between work stations

[igormp]

The copy speed you mentioned is quite close to the speeds reached in your first post from PC1 to PC2 (3~4 Gbit matches the 377 MB/s you got). I'm not sure if the problem lies within the switch or something in your computers.

[TheTwist]

Hmm yeah me neither.  Not sure what other settings can be done with the Switch and i am sure the PC's are setup to how its sure with their 10Gbit card

[jasonvp]

I'd suspect the switch, personally.  Initially I was curious if it was the NIC in your PC because the common 10GigEth NIC in PCs these days is the crap Aquantia.  But, it looks like you have the real deal from Intel, so that's not the thing.

 

My two servers in my basement, hanging off the same Cisco switch:

 

joker$ iperf3 -c 192.168.10.3
Connecting to host 192.168.10.3, port 5201
[clip]
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.34 Gbits/sec    0             sender
[  5]   0.00-10.05  sec  10.9 GBytes  9.29 Gbits/sec                  receiver

 

IOW: I'm pretty much getting 10G/s, and that's not with jumbos enabled, either.  From my Mac up in my office (2nd floor) to one of my servers (basement), meaning it's crossing two switches:

 

harleyquinn$ iperf3 -c 192.168.10.1
Connecting to host 192.168.10.1, port 5201
[clip]
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  10.6 GBytes  9.07 Gbits/sec                  sender
[  4]   0.00-10.00  sec  10.6 GBytes  9.07 Gbits/sec                  receiver

Again: 10G/s or so.

 

I'd be looking into the switch.

[Alex Atkin UK]

On 8/15/2020 at 7:50 PM, jasonvp said:

I'd suspect the switch, personally.  Initially I was curious if it was the NIC in your PC because the common 10GigEth NIC in PCs these days is the crap Aquantia.

What's wrong with Aquantia?

[TheTwist]

@jasonvp yeah been trying to find whats wrong but i cannot see anything that would cause it.  

 

I know you're probably only familiar with Cisco Switches only. But what settings do you have so I can compare. 

 

The Server is LAGed into 4 ports

Frame size for each port is 1500

Flow Control disabled 

 

[igormp]

Another idea popped in my mind: would it be possible for you to connect one device straight to another, skipping the switch entirely? i.e.: workstation 1 to workstation 2, or any of the workstations to the synology NAS.

 

Doing an iperf this way would make sure that the problem doesn't lies in the end devices, but with something in the middle (probably the switch itself).

[jasonvp]

12 minutes ago, TheTwist said:

yeah been trying to find whats wrong but i cannot see anything that would cause it.  

 

I know you're probably only familiar with Cisco Switches only. But what settings do you have so I can compare. 

I don't have any settings that matter, but I'm also not familiar with Netgear switches.  All of my configurations are around LAGs and/or VLANs.  Nothing that's going to affect a single stream of 10Gbit/sec at all.