First FreeNAS Build. Compatibility check and General advice.

[Inrix]

I'm doing my first FreeNAS server build for a new 20TB Array that im planning on setting up and want some advice on if my current plan for it will work and if the parts I am planning on using are compatible with freenas because I know it can be picky... And also just some general advice on how Im planning on doing things...

Sorry for the length of this post <3

The PC/Main Parts:
Im planning on buying this: https://www.trademe.co.nz/computers...00.htm?rsqid=2b500849e2e84349b3f1c699fdea6512 Second hand, it has 6 SATA ports on it which is enough for all the drives I'm planning on using plus enough PCIE slots for network cards, an extra SATA card (https://www.trademe.co.nz/computers...40.htm?rsqid=2313b0555a7c41b099d226387c56a45f) for a boot drive and a TV Tuner Card (https://www.pbtech.co.nz/product/TVNHGR1629/Hauppauge-tunersWinTV-quadHD-New-Model---WinTV-qua)

The Drives/Array Plan:
I currently have 3 4TB WD Blues one of which has a bad sector on it so I won't be using it.
Their current use is for storing a variety stuff:
- My Media Archive so TVShows/Movies/Photos&Videos etc
- My GoogleDrive which is what I use for backing up anything super important.
- A copy of old Server Archives for the community I run.
- A Steam Cache for games that I'm not currently playing that aren't on my SSD's in my main rig.
- Backups of old work I have done (Just more archives).

I currently only have about 700GB free out of the 12TB total between the three drives so I'm planning on getting another 4 of them from here (https://www.pbtech.co.nz/product/HDDWD1940/WD-4TB-Blue-Edition-35-SATA3-Internal-HDD-5400RPM)
and putting them plus my current two that are fine (6 Drives) into a RaidZ setup.

The idea being that it should give me some redundancy (With important stuff being stored on GoogleDrive) and from what I have read around it should let me Read off the disks at about 5-6x the speed of one of the drives and write at 1x the speed of one of the drives. Though I read somewhere that with RaidZ the write speeds are apparently increased as well? So if anyone could clarify that, it would be great, if the write speeds are only the speed of one drive I would likely just get a SSD for a write cache. This should give me roughly a 20TB array.

Generally the only thing I would be using the Array for on a daily basis is watching my Media Collection, Downloading & Watching the daily Floatplane Videos from LTT, Downloading from steam (I plan to setup a steamcache on it so that I can cache all my downloads for quick installs to my SSD on my main rig) and probably some google drive backups too if possible...

Networking:
What I would really love is to be able to have 10Gigabit between the NAS and my Main Rig however they are probably going to be about 15-20m apart from each other and everywhere I looked the cost for two 10Gig cards plus that much cable was nearly 1k NZD... So, I decided that I might try a different approach.

I have fiddled around with SMBMultichannel before and it has worked great allowing me to move files between my server and main rig at up to 3gb/s with 3 nics in each machine, and according to this post: https://forums.freenas.org/index.php?threads/smb-multichannel.54926/ FreeNAS now aparently supports it. With the SATA card and the TV Tuner card I would have about 4 PCIE&PCI slots left which would be enough for me to throw a bunch of https://www.pbtech.co.nz/product/NETTPL3468/TP-Link-TG-3468-32-bit-Gigabit-PCIe-Network-Adapte these into it. Or I might be able to put a single 4Port card in it but it depends on if I can get one I currently have working (Which I'm fairly sure is dead) or if I can take one out of my PfSense Router and use multiple of the above adaptors in that instead.

With either way It will give me another 4 ports on the NAS meaning I should be able to transfer at 5gb/s over the network using SMBMultichannel and just a bunch of Network Cables which I'm happy to have over paying stupid amounts for 10gig. (Though if you know of some cheap 10gig cards/cables that work in FreeNAS and would do a distance of around 15-20m please do tell me)

Software:
This ones kinda complicated... I want to run a bunch of things on the FreeNAS box alongside the main file storage:

Plex (Mainly for organising media but also for TV LiveStreaming & Recording to the NAS with the Receiver Card I mentioned above) < Accessing the 20TB Array
SteamCache (Specifically This: https://github.com/steamcache/generic for caching the installs of games over the network. Mainly so I can keep the SSD's in my main rig clean and whenever I want to play a game I can just install off steam at whatever max speed I can pull from the NAS) < Accessing the 20TB Array
Prometheus (https://prometheus.io/) & Graphana (https://grafana.com/) < Accessing a SSD
Node.js (For several website's/API's and also for the script that downloads daily FloatPlane videos previously mentioned) < Accessing the 20TB Array & a SSD
GoogleDrive

Now I know that I can run VM's and Docker through FreeNAS but I don't know how well this would actually work with everything above in regards to file system access, network access ETC. Im hoping that I will just be able to setup perhaps a single linux VM for all of them and allocate that a nice portion of resources and access to the drives so that stuff like SteamCache can serve stuff to me via all the adapters.

That's kinda it... I'd really like some input as to what I could be doing better or what may or may not work... Or ways that I could improve upon it, and sorry for the length of the post 

[leadeater]

13 minutes ago, Inrix said:

Im planning on buying this: https://www.trademe.co.nz/computers...00.htm?rsqid=2b500849e2e84349b3f1c699fdea6512 Second hand

You are aware the HP Z400 only has 2 HDD bays?

[Inrix]

Just now, leadeater said:

You are aware the HP Z400 only has 2 HDD bays?

Im planning on using a different case/mounting system for it.

[leadeater]

16 minutes ago, Inrix said:

What I would really love is to be able to have 10Gigabit between the NAS and my Main Rig however they are probably going to be about 15-20m apart from each other and everywhere I looked the cost for two 10Gig cards plus that much cable was nearly 1k NZD... So, I decided that I might try a different approach.

You can get much cheaper 10Gb options from ebay if you are prepared to, I buy most of my stuff from there rather than from trademe.

 

https://www.ebay.com/itm/MNPA19-XTR-10GB-Network-Kit-Mellanox-ConnectX-2-10Gbe-NIC-10GBe-1m-SFP-Cable/132480616807?hash=item1ed875c967:g:0hIAAOSwnsRaZ3Fy

[Inrix]

1 minute ago, leadeater said:

You can get much cheaper 10Gb options from ebay if you are prepared to, I buy most of my stuff from there rather than from trademe.

 

https://www.ebay.com/itm/MNPA19-XTR-10GB-Network-Kit-Mellanox-ConnectX-2-10Gbe-NIC-10GBe-1m-SFP-Cable/132480616807?hash=item1ed875c967:g:0hIAAOSwnsRaZ3Fy

That would be awesome to link between the NAS and my router. But I would need a much longer cable to get from the NAS to my main pc... I spent ages looking around on ebay yesterday to try and find a longer fiber one and couldnt  And the cheapest 10gig rj45 adapter I could find was like 200$ nzd... Any idea's?

[Inrix]

4 minutes ago, leadeater said:

-Snip-

Oh also I dont know if FreeNAS supports that card  That was kinda why I ended up considering going with a SMBMultichannel setup because the only "supported" cards I could find were much more expensive... Do you know if FreeNAS would actually work with those cards you linked?

[leadeater]

1 minute ago, Inrix said:

That would be awesome to link between the NAS and my router. But I would need a much longer cable to get from the NAS to my main pc... I spent ages looking around on ebay yesterday to try and find a longer fiber one and couldnt  And the cheapest 10gig rj45 adapter I could find was like 200$ nzd... Any idea's?

Best bet would be to use optical and fibre SFP+, @Windows7ge did you do this or was it someone else?

 

Two of these:

https://www.ebay.com/itm/MNPA19-XTR-10GB-MELLANOX-CONNECTX-2-PCIe-X8-10Gbe-SFP-Network-Card-RT8N1/142658266831?hash=item21371866cf:g:Nw8AAOSwJWFaXi5m

https://www.ebay.com/itm/10GBE-For-Mellanox-ConnectX-2-Single-Port-SFP-Network-Card-MNPA19-XTR-transceiv/161799895137?hash=item25ac067861:g:47UAAOSwyQtVxS8n

 

And a premade length of multimode OM3/OM4

[Inrix]

7 minutes ago, leadeater said:

-Snip-

Do you know where I could get the fiber from/how expensive it is for say 15m? I've never done SFP+ stuff so sadly I'm not that well informed on how to set it all up/what to look for when getting parts. Id love to be able to use it though, I just couldn't find any fiber that wasn't under like 300+ NZD for 15m

[leadeater]

5 minutes ago, Inrix said:

Do you know where I could get the fiber from/how expensive it is for say 15m? I've never don't SFP+ stuff so sadly I'm not that well informed on how to set it all up/what to look for when getting parts. Id love to be able to use it though, I just couldn't find any fiber that wasn't under like 300+ NZD for 15m

Ebay .

 

https://www.ebay.com/itm/15M-10G-OM3-Armored-Cable-Fiber-Patch-Cord-LC-to-LC-3-0mm-MM-50-125-Duplex/112622569406?hash=item1a38d407be:g:Qi4AAOSwNWxZ9rgm

 

Also yes I'm fairly sure they are supported, older versions of FreeNAS didn't support them so rather mixed information online about it. If you are worried about it you can step up to Intel X520-SR2 or if you want simple Intel X540-T1/2.

[Inrix]

3 minutes ago, leadeater said:

-Snip-

O, So would I just buy two of these:

https://www.ebay.com/itm/MNPA19-XTR-10GB-MELLANOX-CONNECTX-2-PCIe-X8-10Gbe-SFP-Network-Card-RT8N1/142658266831?hash=item21371866cf:g:Nw8AAOSwJWFaXi5m

and these:

https://www.ebay.com/itm/10GBE-For-Mellanox-ConnectX-2-Single-Port-SFP-Network-Card-MNPA19-XTR-transceiv/161799895137?hash=item25ac067861:g:47UAAOSwyQtVxS8n

 

And then I can just plug this fiber cable

https://www.ebay.com/itm/15M-10G-OM3-Armored-Cable-Fiber-Patch-Cord-LC-to-LC-3-0mm-MM-50-125-Duplex/112622569406?hash=item1a38d407be:g:Qi4AAOSwNWxZ9rgm

into them? Or would I need to fiddle more with the fiber?

[leadeater]

1 minute ago, Inrix said:

O, So would I just buy two of these:

https://www.ebay.com/itm/MNPA19-XTR-10GB-MELLANOX-CONNECTX-2-PCIe-X8-10Gbe-SFP-Network-Card-RT8N1/142658266831?hash=item21371866cf:g:Nw8AAOSwJWFaXi5m

and these:

https://www.ebay.com/itm/10GBE-For-Mellanox-ConnectX-2-Single-Port-SFP-Network-Card-MNPA19-XTR-transceiv/161799895137?hash=item25ac067861:g:47UAAOSwyQtVxS8n

 

And then I can just plug this fiber cable

https://www.ebay.com/itm/15M-10G-OM3-Armored-Cable-Fiber-Patch-Cord-LC-to-LC-3-0mm-MM-50-125-Duplex/112622569406?hash=item1a38d407be:g:Qi4AAOSwNWxZ9rgm

into them? Or would I need to fiddle more with the fiber?

Correct, although I'd buy two cables if you can in case one get's damaged during shipping. Return times and replacements is really long from NZ.

 

Working with SFP+ and premade fibre is pretty simple, you do need to be a bit more careful compared to Cat5/6 and not touch the ends of the connectors. Just plug the NICs in, slide in the SFP+ modules (the correct orientation) then plug the cable in.

[Inrix]

Just now, leadeater said:

-Snip-

Awesome! Thanks so much, I'll probably do that then! Are the fiber cables very delicate? Like would it be a good idea to try run it across the ceiling versus having it run along the skirting? Itl probably have to go under  the door to my room but I could likely pad it with something to protect it.

[leadeater]

Just now, Inrix said:

Awesome! Thanks so much, I'll probably do that then! Are the fiber cables very delicate? Like would it be a good idea to try run it across the ceiling versus having it run along the skirting? Itl probably have to go under  the door to my room but I could likely pad it with something to protect it.

The one I pointed to is an armored one so not fragile, do treat it with care though. The ends of it are not armored and the connectors are just plastic so don't pull on them hard. Dust is the real problem with these cables, always put the dust caps back on the ends if it's not plugged in.

[Inrix]

1 minute ago, leadeater said:

-Snip-

Sweet. If I needed to try and find a cable with a different length what should I look for/search in ebay? I was looking for ages last night with a friend trying to find cables but couldnt find anything 

[leadeater]

2 minutes ago, Inrix said:

Sweet. If I needed to try and find a cable with a different length what should I look for/search in ebay? I was looking for ages last night with a friend trying to find cables but couldnt find anything 

Xm OM3

 

 

 

Edit:

Oh and make sure it's LC-LC

 

[Windows7ge]

1 hour ago, leadeater said:

Best bet would be to use optical and fibre SFP+, @Windows7ge did you do this or was it someone else?

That was me. Wrote a guide about it. Hey OP, cheap effective multi-mode OM4 fibre cables can be bought off www.fs.com. Precut lengths are made up to 15M (49ft) but you can request custom length. Using 2 SFP-10GSR-85 transceivers will allow you to use a cable up-to 300M (984ft) @ 10Gbit full duplex.

 

Also like leadeater mentioned do be gentle with the cables. Fiber IS strong but it's still glass (or Zirconia Ceramic Furul...something something...it's glass/ceramic...depending on who made it) and being too rough with it like anything else can break it. Also yes. Dust caps. Keep em on both the cable and the rubber inserts on the transceivers when not in use.

 

Edit: Also, don't stare into the fiber once powered on with one end plugged in. The specific transceiver I mentioned is a 1 watt 850nm wavelength laser diode. It's worse for your eyes than staring into a laser pointer and that's already really bad.