Jump to content

10Gbps networking question for Linux folks

road hazard
By road hazard
in Networking
 Share
Posted

I have a media server running Mint 19.2 that is always on and has a bunch of 4TB drives in a MDADM RAID 6 array (md0). In that same server, I have a bunch of 8TB drives in a MDADM RAID 5 array (mdbackup) that the RAID 6 array is backed up to. I want to move the backup array to a separate server in case of some major hardware failure (water damage, falling off the table, etc). Because right now, if a pipe leaks onto my main server and fries the backplane or something, I'd love EVERYTHING. I want to physically separate the arrays.

 

So, I'm in the process of piecing together another server. The amount of data I have is in the 35TB range. To do a full restore, over a 1GB network, it would take days and days. If I switch to 10Gbps, it would only be hours. My MDADM array can write at around 500MB/s so a 10GB network would be perfectly fine.

 

BUT, to keep cost low, could I get away with just buying some 10Gbps network cards and using a DAC connection between them (eliminating the need for 10Gbps SFP+ switch)? The only 2 PCs on my network that would need 10Gbps speed are the main media server and the backup one when they are talking to each other but, I'll still need both to be able to route out to the internet on their 1G NICs that will plug into my 10/100/1000 switch. Problem is, I don't know how to accomplish any of that. Is this easily doable or should I just pick up a switch that has 16 or 24, 10/100/1000 ports plus 2, SFP+ ports?

 

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
7 hours ago, road hazard said:

BUT, to keep cost low, could I get away with just buying some 10Gbps network cards and using a DAC connection between them (eliminating the need for 10Gbps SFP+ switch)?

Yes, you can DAC your two 10Gb/s network cards together to transfer files without a switch. This is how I have my PC connected to my server (Windows to ESXi).

You will have to manually assign IP addresses / subnet mask, but it is pretty easy.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
16 hours ago, scottyseng said:

Yes, you can DAC your two 10Gb/s network cards together to transfer files without a switch. This is how I have my PC connected to my server (Windows to ESXi).

You will have to manually assign IP addresses / subnet mask, but it is pretty easy.

Not to sound dumb but....... I'm dumb when it comes to networking.

 

That's great news that I can maybe pull this off without a 10Gbps switch. If you can break down what I need to do, as if you're talking to a 5 year old, I'd greatly appreciate it! In the end, server A and server B will both have 1G and 10Gbps NIC. I need both to be able to get out to the internet over their 1G NICs (easy enough... just plug their 1G NICs into the switch) but when they talk to each other, I need all communication to go over the 10Gbps NICs.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
On 11/25/2019 at 1:27 PM, road hazard said:

Not to sound dumb but....... I'm dumb when it comes to networking.

 

That's great news that I can maybe pull this off without a 10Gbps switch. If you can break down what I need to do, as if you're talking to a 5 year old, I'd greatly appreciate it! In the end, server A and server B will both have 1G and 10Gbps NIC. I need both to be able to get out to the internet over their 1G NICs (easy enough... just plug their 1G NICs into the switch) but when they talk to each other, I need all communication to go over the 10Gbps NICs.

Im guessing the media server is headless, you need to figure out the name of each interface and give it a static ip using netplan. here is a tutorial (I haven't used that tutorial but I have done it before) https://linuxconfig.org/how-to-configure-static-ip-address-on-ubuntu-19-04-disco-dingo-linux

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
21 hours ago, mtz_federico said:

Im guessing the media server is headless, you need to figure out the name of each interface and give it a static ip using netplan. here is a tutorial (I haven't used that tutorial but I have done it before) https://linuxconfig.org/how-to-configure-static-ip-address-on-ubuntu-19-04-disco-dingo-linux

I thought about the headless route but since I'm still a bit of a newbie with Linux, I figured it would be better to have a nice and fancy GUI for things. Thanks for the tutorial! Once I get the 10Gbps NICs, I'll give it a shot. If all else fails, I'll give in and just buy a 10Gbps switch. :). Probably a Dell PowerConnect 5524 off eBay.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
2 minutes ago, road hazard said:

I thought about the headless route but since I'm still a bit of a newbie with Linux, I figured it would be better to have a nice and fancy GUI for things. Thanks for the tutorial! Once I get the 10Gbps NICs, I'll give it a shot. If all else fails, I'll give in and just buy a 10Gbps switch. :). Probably a Dell PowerConnect 5524 off eBay.

No problem. That tutorial should also work if it is not headless but you can also set a static ip in setthings on the GUI easilly.

btw if you use the tutorial I recommend that you backup the files that you edit in the tutorial just in case it doesn't work you can replace them

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
On 11/27/2019 at 12:52 PM, mtz_federico said:

No problem. That tutorial should also work if it is not headless but you can also set a static ip in setthings on the GUI easilly.

btw if you use the tutorial I recommend that you backup the files that you edit in the tutorial just in case it doesn't work you can replace them

I think I just need a few more things answered before I buy everything........ I'm probably going to go with Mellanox MNPA19-XTR Connect X-2 cards. So long as I can hit about 300-500MB/s (the max speed of my RAID 6 array), I'm good.

 

Now, I know I can install the 10Gbps cards into my main server and backup server and join them with a DAC cable (to talk to each other) and use their 1Gbps NICs to attach to the switch and get to the internet and talk to other devices on the network (thus, avoiding the need for a 10Gbps switch) but..... lets say I can't figure out the routing issues and want to just hook the 10Gbps NICs into the Dell 5524, along with all my other devices,........... what kind of cable goes from the 10Gbps NICs into the switch? At that point, do I buy, 4, fiber optic SFP+ transceivers for the switch and 10Gbps NICs and a few feet of fiber optic cable or do I buy just TWO DAC cables and hook each 10Gbps NIC into the switch that way?

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

you will lower your cost significantly going to 10 gbE. You wont need modules and some switches have 10 gbE ports.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

a switch like, https://www.amazon.ca/2x10-Gig-Multi-Gig-Lifetime-Protection-GS110EMX/dp/B0765ZPY18/ref=sr_1_3?keywords=10G%2Bswitch&qid=1575152973&s=electronics&smid=A3DWYIK6Y9EEQB&sr=1-3&th=1 has two ports for each of your media servers the rest are 1G.

 

you could get two of these 

https://www.newegg.com/rosewill-rc-nic412v2/p/N82E16833166130?Description=10gb ethernet&cm_re=10gb_ethernet-_-33-166-130-_-Product

 

some cat6 cables,

https://www.newegg.com/black-coboc-7-ft-network-ethernet-cables/p/N82E16812117402?Description=cat6 cables&cm_re=cat6_cables-_-12-117-402-_-Product

 

and your done

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
1 hour ago, tech.guru said:

a switch like, https://www.amazon.ca/2x10-Gig-Multi-Gig-Lifetime-Protection-GS110EMX/dp/B0765ZPY18/ref=sr_1_3?keywords=10G%2Bswitch&qid=1575152973&s=electronics&smid=A3DWYIK6Y9EEQB&sr=1-3&th=1 has two ports for each of your media servers the rest are 1G.

 

you could get two of these 

https://www.newegg.com/rosewill-rc-nic412v2/p/N82E16833166130?Description=10gb ethernet&cm_re=10gb_ethernet-_-33-166-130-_-Product

 

some cat6 cables,

https://www.newegg.com/black-coboc-7-ft-network-ethernet-cables/p/N82E16812117402?Description=cat6 cables&cm_re=cat6_cables-_-12-117-402-_-Product

 

and your done

The Dell 5524 (if I go the route of needing a switch) is AROUND $80-90 on eBay. The Mellanox cards are about $40 each. Your cable price is cheap but unless I'm missing something, the switch and NICs you reference make that route more expensive.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

I'd be very interested to hear how this works in the end because I'm finding backing up over the network to be hugely bottlenecked by IO limitations of the network itself.  I can't even max out a HDD over 2.5Gbit let alone an SSD.

I don't bother with RAID instead doing a 1:1 backup from the NAS HDD to USB HDDs.  I figure its quicker to recover than having to repair a RAID plus no issues with backplane failure, SATA/SAS chipset failure, as I can easily just swap everything to a standard motherboard.

Router:  Intel N100 (pfSense) WiFi6: Zyxel NWA210AX (1.7Gbit peak at 160Mhz)
WiFi5: Ubiquiti NanoHD OpenWRT (~500Mbit at 80Mhz) Switches: Netgear MS510TXUP, MS510TXPP, GS110EMX
ISPs: Zen Full Fibre 900 (~930Mbit down, 115Mbit up) + Three 5G (~800Mbit down, 115Mbit up)
Upgrading Laptop/Desktop CNVIo WiFi 5 cards to PCIe WiFi6e/7

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
On 11/30/2019 at 10:40 PM, Alex Atkin UK said:

I'd be very interested to hear how this works in the end because I'm finding backing up over the network to be hugely bottlenecked by IO limitations of the network itself.  I can't even max out a HDD over 2.5Gbit let alone an SSD.

I don't bother with RAID instead doing a 1:1 backup from the NAS HDD to USB HDDs.  I figure its quicker to recover than having to repair a RAID plus no issues with backplane failure, SATA/SAS chipset failure, as I can easily just swap everything to a standard motherboard.

Will let you know! I have some Mellanox cards and a DAC cable arriving in a few days. I decided to skip on using the switch and just connecting to the two servers via DAC, leaving their normal 1Gbps NICs for internet/LAN traffic. I think I mentioned somewhere else that my RAID 6 (MDADM) array maxes out around 300-500MB/s. A 10Gbps connection is capable of (theoretically) 1,250MB/s. So..... if I can get at least 1/2 the max of a 10Gbps connection (which I'm sure will not be a problem), I'll be a happy camper!

 

Heck, at that point, my I/O sub-system will be the bottleneck and I'm fine with that. For now. :)

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
On 11/30/2019 at 10:40 PM, Alex Atkin UK said:

I'd be very interested to hear how this works

 

Quick update. Did some iperf testing this morning and here are the results. (I'm no iperf expert so this info might be redundant but I'll include text from the server and client.)

 

Server:

  

iperf -s -B 192.168.10.1
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 192.168.10.1
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.10.1 port 5001 connected with 192.168.10.2 port 48749
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  10.9 GBytes  9.39 Gbits/sec
[  4] local 192.168.10.1 port 5001 connected with 192.168.10.2 port 48767
------------------------------------------------------------
Client connecting to 192.168.10.2, TCP port 5001
Binding to local address 192.168.10.1
TCP window size: 2.68 MByte (default)
------------------------------------------------------------
[  6] local 192.168.10.1 port 36691 connected with 192.168.10.2 port 5001
[  6]  0.0-10.0 sec  9.97 GBytes  8.56 Gbits/sec
[  4]  0.0-10.0 sec  8.62 GBytes  7.39 Gbits/sec
[  4] local 192.168.10.1 port 5001 connected with 192.168.10.2 port 57933
[  4]  0.0-10.0 sec  10.9 GBytes  9.39 Gbits/sec
------------------------------------------------------------
Client connecting to 192.168.10.2, TCP port 5001
Binding to local address 192.168.10.1
TCP window size: 2.14 MByte (default)
------------------------------------------------------------
[  4] local 192.168.10.1 port 40939 connected with 192.168.10.2 port 5001
[  4]  0.0-10.0 sec  10.9 GBytes  9.33 Gbits/sec
[  4] local 192.168.10.1 port 5001 connected with 192.168.10.2 port 45261
[  4]  0.0-60.0 sec  65.6 GBytes  9.40 Gbits/sec

From the client:

  

iperf -c 192.168.10.1 -B 192.168.10.2
------------------------------------------------------------
Client connecting to 192.168.10.1, TCP port 5001
Binding to local address 192.168.10.2
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.2 port 48749 connected with 192.168.10.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.9 GBytes  9.40 Gbits/sec

iperf -c 192.168.10.1 -B 192.168.10.2 -d
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 192.168.10.2
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.10.1, TCP port 5001
Binding to local address 192.168.10.2
TCP window size: 1.96 MByte (default)
------------------------------------------------------------
[  5] local 192.168.10.2 port 48767 connected with 192.168.10.1 port 5001
[  4] local 192.168.10.2 port 5001 connected with 192.168.10.1 port 36691
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  8.62 GBytes  7.40 Gbits/sec
[  4]  0.0-10.0 sec  9.97 GBytes  8.56 Gbits/sec

iperf -c 192.168.10.1 -B 192.168.10.2 -r
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 192.168.10.2
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.10.1, TCP port 5001
Binding to local address 192.168.10.2
TCP window size: 2.73 MByte (default)
------------------------------------------------------------
[  5] local 192.168.10.2 port 57933 connected with 192.168.10.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  10.9 GBytes  9.40 Gbits/sec
[  4] local 192.168.10.2 port 5001 connected with 192.168.10.1 port 40939
[  4]  0.0-10.0 sec  10.9 GBytes  9.32 Gbits/sec

iperf -c 192.168.10.1 -B 192.168.10.2 -t 60
------------------------------------------------------------
Client connecting to 192.168.10.1, TCP port 5001
Binding to local address 192.168.10.2
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.2 port 45261 connected with 192.168.10.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  65.6 GBytes  9.40 Gbits/sec

 

Also including a 'during' and 'after' screenshot from System Monitor. (You'll see some CPU spiking in the 'after' picture. I don't think that was the 10Gbps traffic (which did consume a tiny bit of CPU power) but this server runs Plex and Emby and Emby was doing SOMETHING and I think that is what caused the random CPU spikes during the end.

 

Long story short, it looks like I'm getting the promised speed. Unfortunately, my I/O subsystem can only write to my RAID 6 array at around 300MB/s sustained. It will spike to 500MB/s for small'ish files but when copying 15, 20 gig files...... sorta tops out around 300+MB/s. :( Because of that, I don't see a need to tweak or fine tune anything since I can't write it any faster. lol  Although..... maybe I'll look into using an SSD to cache things?!?!?!? But, I don't think MDADM supports caching. :(

 

Oh well. Beats 110MB/s over 1 gig. :)

 

I didn't use a switch. Just a passive DAC cable between the servers and edited their hosts file to point to each other. Both servers' 10Gbps NICs are using static IPs and subnet masks (no DNS or gateway entries) and they can both get out on the internet using their 1Gbps NICs but when talking to each other, all traffic flows at 10Gbps! Well, maybe 3Gbps. hahaha

 

The cards were $40 each and $30 for the DAC cable.

 

 

during.png

after.png

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Well call me silly. Just discovered something...... after some more testing copying files to/from the RAID 6 array (using a new SSD), it appears my RAID 6 array is capable of writing at 500MB/s (about the max read speed of my new SSD). I did some benchmarking on my backup server and the SSD in there is capable of reading at around 550MB/s. So..... why can I only write to my RAID 6 array (over 10Gbps connection) at 300'ish MB/s? I tried setting the MTU to 9000 and that got me a slight speed bump then xfers started crawling at 70MB/s so I reverted the change.

 

What can I look into to get more speed over the DAC?

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

I just got done conducting another test. I pulled the Linux boot SSD from server A and replaced it with another SSD (rated for over 500MB/s read/write) and installed Windows 10 on there and left the 10Gbps NIC in. I pulled the 2nd 10Gbps NIC out of the other Linux Mint box and placed it into my Windows 10 PC (which also has a SSD capable of 500+MB/s read/write). I copied about 50 gigs worth of data over the 10Gbps connection (Windows 10 on both PCs) and........... I was maxing out my SSD and getting about 480-500MB/s!!! This was with default Windows 10 settings (1500 MTU, etc. No tweaking.) So, looks like my 10Gbps cards and DAC cable are 100% fine..... it's a software tuning problems with Linux that needs figuring out.

Link to comment
Share on other sites
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing Networking

×