SFP+ Compatibility

[Dschulz72]

Hey all. I'm fairly new to the NAS and home server world. I've been running Unraid on my server for about a year now, and I'm starting to add more services to my machine. 

Since I'm adding more services/devices and I just moved into a new place, I intended to upgrade the link from my unraid box to my switch to 10GbE. So I bought a Dell Poweredge 6224 switch and a Mellanox ConnectX-3 SFP+ NIC. I wired them up using a Dell active DAC cable and just nothing. No lights, no connection, nothing. I think I bought the wrong cable to connect them, but I'm not sure. I did order a passive cable that should be here in a day or so. Documentation on this stuff is pretty dense.

 

Anyone out there know what I'm doing wrong? I realize SFP/SFP+/QSFP isn't as straightforward as RJ45. 

[Lurick]

Did you configure the ports? I'm pretty sure the dell ports need to be brought up and are shutdown by default.

[Dschulz72]

If I select the port in the web interface it shows the link as 'Down' but the admin status is 'Enable'.

[jaslion]

10 minutes ago, Dschulz72 said:

If I select the port in the web interface it shows the link as 'Down' but the admin status is 'Enable'.

Enable or enabled? If the first then they are still off

[Dschulz72]

It says 'Enable', not 'Enabled'. It is a drop-down; the options are just 'Enable' and 'Disable'. Would it be fair to wait for the new (passive) DAC cable?

[Jarsky]

I wouldn't even bother with SFP+ really with UnRAID, unless youre writing/reading a lot from cache pools. 

[Dschulz72]

I could see a scenario in the near future where this server is simultaneously:

1. Running/streaming a VM to a LAN client (from the NVME cache)

2. Running a 4K Plex stream to a LAN client

3. Handling 5 security camera streams to a separate cache pool

I thought this would be an appropriate time to try 10Gb, especially since I'm doing CAT6 runs though my house, but link aggregation would be another option. 

 

I'll also note that the Unraid box does seem to see and properly identify the Mellanox card. 

[LIGISTX]

4 hours ago, Dschulz72 said:

I could see a scenario in the near future where this server is simultaneously:

1. Running/streaming a VM to a LAN client (from the NVME cache)

2. Running a 4K Plex stream to a LAN client

3. Handling 5 security camera streams to a separate cache pool

I thought this would be an appropriate time to try 10Gb, especially since I'm doing CAT6 runs though my house, but link aggregation would be another option. 

 

I'll also note that the Unraid box does seem to see and properly identify the Mellanox card. 

The “issue” with unraid is, it’s really just not set up to be performant like this. Unraid is a nice simple solution for home users who want to have a fancy JBOD. Something like ZFS (truenas) would actually be able to perform fast enough to make faster networking more worth while. 
 

With that said, remote access to a VM takes effectively no bandwidth, you are basically just streaming video. 4k Plex is the same, the biggest file you will ever have is a full bluray rip; I don’t think any of mine are more then ~80Mb/s, which is not even 1/10th of what gigabit will do. And some IP cameras, unless your recording them all in uncompressed 4k (your not), won’t be an issue either, seeing as a full HDR bluray with full atmos audio stream is only ~80Mbps, IP cameras are in the single, maybe just breaking into double digit Mbps.

 

You probably don’t really need nvme cache, or even SSD cache for much of this. VM’s should certainly live on SSD, nvme > sata of course, but it certainly is not needed and will not really help much.

 

The “issue” (but also massive benefit depending on your outlook) is that unraid is not actually RAID; it’s a “fancy JBOD with redundancy”. This means you really don’t get speed increases like you do with RAID, and where ZFS has ARC, unraid does not (you can add cache drives, which is how people speed up their pools, but honestly reading out home user data from harddrives with no cache can typically saturate gigabit anyways, and attempting to write to harddrives in any sort of random way over a network is just going to be a not fun time)

 

You should certainly use SSD’s for running rocker containers and VM’s, but probably don’t need to store your IP camera data onto SSD, that should just go to spinning rust. And while 10GbE is nice, it likely won’t make much of a difference with an unraid setup for the reasons listed above; and none of the use cases really “need” it anyways. Obviously who am I to say faster isn’t better; I am looking into 10GbE myself for a direct connection between PC and Truenas, but I’d like to start editing my photos directly off my NAS, with a full SSD ZFS vdev over iSCSI instead of your normal SMB. For that use case, I have an argument for myself to actually do it. 

[Dschulz72]

All very good points, @LIGISTX. 

I should clarify that the IP cameras are writing to two WD Purple Drives, they've just been placed in a 'cache pool' in Unraid, separate from the main array. 

The main array has 2x NVME 1.92TB enterprise drives as a cache already. If I'm being honest, I also just want to see numbers above 1Gbps on file copies. XD Especially because I already have the hardware (in theory) to make it happen. I might explore TrueNAS/ZFS in a future adventure if I feel like Unraid isn't meeting my needs. 

 

So even if it's not super necessary, I think it's fun to play with hardware, and maybe I'll find a better use for it sometime down the line. 

[LIGISTX]

2 hours ago, Dschulz72 said:

All very good points, @LIGISTX. 

I should clarify that the IP cameras are writing to two WD Purple Drives, they've just been placed in a 'cache pool' in Unraid, separate from the main array. 

The main array has 2x NVME 1.92TB enterprise drives as a cache already. If I'm being honest, I also just want to see numbers above 1Gbps on file copies. XD Especially because I already have the hardware (in theory) to make it happen. I might explore TrueNAS/ZFS in a future adventure if I feel like Unraid isn't meeting my needs. 

 

So even if it's not super necessary, I think it's fun to play with hardware, and maybe I'll find a better use for it sometime down the line. 

I admit I’m not very familiar with how unraid works. I don’t know why you would put spinning rust as a cache pool, and how it isn’t going to try and actually use that as cache for your main array. Or why you wouldn’t just add drives to the main array and save the IP cam footage to the main array. But, I’m sure there is a reason, just a bit strange to me. 
 

And I get it, I definitely don’t need the homelab hardware I have, but it sure is fun to play with.