NAS + Router Suggestions

[h0m3]

Hello friends,

I'm planning on building a proper NAS setup on my network. Today I have a Dell SFF with a i5 10500T. Its a good computer but the realtek nic has its problems, theres little room to ram upgrade since its only two sodimm slots and the hard drivers run over USB on external cases. I've already have some parts today.

My plan is the follow:
- AMD Ryzen 5700G
- MSI  B550m Pro Dash (already have)

- 128GB DDR4 (4 x 32) - I've found a SK Hynix ECC kit very cheap, probably removed from servers

- A Dell PERC H200 (on the x16 slot)

- A Intel 82576 Quad (on the x1)

- A Intel 82576 Dual (on the other x1)

- 2 Kingston NV1 500GB on the M2 slots

- A Corsair CX750 (already have)

- A CM NR200 or a Corsair 270R (already have both)

I already have two 8TB WD Red drives on my NAS (on ZFS RAID1). They are a bit old and my plan is to replace it with 8 1TB SATA SSDs in the future and run them on RAIDZ1. Today I have cloud backup, data loss is not tragic but its a hassle so my plan is to replace the drives before they die. Unfortunately i'll have to delay this because my budget cannot allow both system and drives. For the time being I plan to keep my 8TB disks.

Today my system has 32GB of RAM but due to the amount of services running its becoming to be limiting, also today I run everything over LXC containers, but some things I would prefer to run in VMs because I can run things that aren't Linux based like pfSense and TrueNAS.

What do I run today:

- ZFS as stated above

- PiHole DNS

- Home Assistant

- The Tplink Omada Controller software

- Nextcloud (most for notes and news)

- A smb + sftp server

- Syncthing

-  A encrypted backup software

A lot of RAM is used to cache ZFS reads, and even if I lower the ZFS cache while those services are pretty lightweight when you add them all together I run everything within a couple gigabytes of full RAM usage. CPU usage is quite low, although the 5700G would be a upgrade I'm fine with anything modern and 6 or more cores. The 8 core was choose only because has more room to run VMs.

I also plan to run pfSense (virtualized) and replace my router, thats why i've added some NICs to the budget. Also I want to runaway from realtek crap hahaha. I checked out the used enterprise market but here the market is very bad so the only things fitting my budget are DDR3 era, waste a lot of power and dont do much like Xeons X5550. Also DDR3 RAM is stupidly expensive, specially ECC that is a requirements on those systems for high density.

What do you guys think? Any suggestions?

[FK FOSS it is Socialism]

2 quick notes

 

fore ryzen and ecc

 

 

and for ram, be sure it's unbuffered

[h0m3]

24 minutes ago, FK FOSS it is Socialism said:

2 quick notes

 

fore ryzen and ecc

 

 

and for ram, be sure it's unbuffered

Thats a extremely good point.

Thanks for noting it. I've almost bought an incompatible kit. Its unbuffered. ECC was a plus, i don't mind running without but this kit is cheaper than non-ECC memory. Now I have to decide between a more expensive RAM kit or a non-APU Ryzen that wont have a graphics output...

I'll probably go with the non-ECC RAM but if you have some suggestions I would appreciate.

[FK FOSS it is Socialism]

the pro APUs support ecc, but no idea if there is a pro of the 5700g

 

on the truenas forums the consensus is very much that you should use ecc for zfs, here you mostly read it's not needed, only you can decide how important your data is to you. so to make an educated decision, i would recommend to educate yourself on that topic =P

[LIGISTX]

8 hours ago, h0m3 said:

Hello friends,

I'm planning on building a proper NAS setup on my network. Today I have a Dell SFF with a i5 10500T. Its a good computer but the realtek nic has its problems, theres little room to ram upgrade since its only two sodimm slots and the hard drivers run over USB on external cases. I've already have some parts today.

My plan is the follow:
- AMD Ryzen 5700G
- MSI  B550m Pro Dash (already have)

- 128GB DDR4 (4 x 32) - I've found a SK Hynix ECC kit very cheap, probably removed from servers

- A Dell PERC H200 (on the x16 slot)

- A Intel 82576 Quad (on the x1)

- A Intel 82576 Dual (on the other x1)

- 2 Kingston NV1 500GB on the M2 slots

- A Corsair CX750 (already have)

- A CM NR200 or a Corsair 270R (already have both)

I already have two 8TB WD Red drives on my NAS (on ZFS RAID1). They are a bit old and my plan is to replace it with 8 1TB SATA SSDs in the future and run them on RAIDZ1. Today I have cloud backup, data loss is not tragic but its a hassle so my plan is to replace the drives before they die. Unfortunately i'll have to delay this because my budget cannot allow both system and drives. For the time being I plan to keep my 8TB disks.

Today my system has 32GB of RAM but due to the amount of services running its becoming to be limiting, also today I run everything over LXC containers, but some things I would prefer to run in VMs because I can run things that aren't Linux based like pfSense and TrueNAS.

What do I run today:

- ZFS as stated above

- PiHole DNS

- Home Assistant

- The Tplink Omada Controller software

- Nextcloud (most for notes and news)

- A smb + sftp server

- Syncthing

-  A encrypted backup software

A lot of RAM is used to cache ZFS reads, and even if I lower the ZFS cache while those services are pretty lightweight when you add them all together I run everything within a couple gigabytes of full RAM usage. CPU usage is quite low, although the 5700G would be a upgrade I'm fine with anything modern and 6 or more cores. The 8 core was choose only because has more room to run VMs.

I also plan to run pfSense (virtualized) and replace my router, thats why i've added some NICs to the budget. Also I want to runaway from realtek crap hahaha. I checked out the used enterprise market but here the market is very bad so the only things fitting my budget are DDR3 era, waste a lot of power and dont do much like Xeons X5550. Also DDR3 RAM is stupidly expensive, specially ECC that is a requirements on those systems for high density.

What do you guys think? Any suggestions?

I bet you could scale down your truenas RAM usage a lot and be totally fine… I ran my homelab on 28GB of ram for years. It hosted truenas (with 10x4 TB in Z2, I gave truenas 16GB of RAM), multiple Ubuntu server VM’s (1-2.5 GB of ram per VM), windows LTSC (2 GB), home assistant VM (1.2 GB), and some docker containers in one of the Ubuntu VM’s… as well as Plex in one of the Ubuntu VM’s, and everything was happy as could be. 
 

Also, I wouldn’t plan on going to SSD based ZFS array… there is just no point in doing so, and will cost you way more money for much less space then normal drives would. Additionally, I wouldn’t worry much about your drives getting old and needed to be replaced, that’s sort of the point of ZFS scrubs and RAID. If a drive starts throwing SMART errors, or if you ever get an error from a scrub, then you need to buy a replacement ASAP, but I wouldn’t worry about swapping drives out “just to be safe”. They can last a very, very long time. 
 

And as someone who has pfsense virtualized… just make sure you understand what your getting into. It’s honestly a better solution to just buy either a netgate (pfsense parent company) box, or buy one of those little mini PC’s (super small, use SODIMM ram, soldered on Atom class CPU’s) as a firewall/router appliance. Virtualized networking infrastructure is fine, just make sure you do know what your doing, and understand your failure modes so when your homelab does go down either for hardware maintenance or due to an issue, you already have a mental picture and plan on how to get at least basic networking back… troubleshooting issues is extra horrible without internet   

[LIGISTX]

1 minute ago, FK FOSS it is Socialism said:

on the truenas forums the consensus is very much that you should use ecc for zfs, here you mostly read it's not needed

It’s really not required. It’s a good idea… especially since the point of ZFS is data resiliency at all costs. But….. it’s not needed. There is no scrub or death, your data will be ok (especially if you have a Z2 or greater array, the more drives you have of parity the easier ZFS’s job is at determining who is lying). Plus, Z1 is “dead” with todays drive sizes and costs per drive anyways, no one should be deploying Z1 these days, Z2 really should be the minimum. 

[h0m3]

10 hours ago, FK FOSS it is Socialism said:

the pro APUs support ecc, but no idea if there is a pro of the 5700g

 

on the truenas forums the consensus is very much that you should use ecc for zfs, here you mostly read it's not needed, only you can decide how important your data is to you. so to make an educated decision, i would recommend to educate yourself on that topic =P

I coudn't find anything about a Pro 5700G either.

[h0m3]

11 hours ago, LIGISTX said:

I bet you could scale down your truenas RAM usage a lot and be totally fine… I ran my homelab on 28GB of ram for years. It hosted truenas (with 10x4 TB in Z2, I gave truenas 16GB of RAM), multiple Ubuntu server VM’s (1-2.5 GB of ram per VM), windows LTSC (2 GB), home assistant VM (1.2 GB), and some docker containers in one of the Ubuntu VM’s… as well as Plex in one of the Ubuntu VM’s, and everything was happy as could be. 
 

Also, I wouldn’t plan on going to SSD based ZFS array… there is just no point in doing so, and will cost you way more money for much less space then normal drives would. Additionally, I wouldn’t worry much about your drives getting old and needed to be replaced, that’s sort of the point of ZFS scrubs and RAID. If a drive starts throwing SMART errors, or if you ever get an error from a scrub, then you need to buy a replacement ASAP, but I wouldn’t worry about swapping drives out “just to be safe”. They can last a very, very long time. 
 

And as someone who has pfsense virtualized… just make sure you understand what your getting into. It’s honestly a better solution to just buy either a netgate (pfsense parent company) box, or buy one of those little mini PC’s (super small, use SODIMM ram, soldered on Atom class CPU’s) as a firewall/router appliance. Virtualized networking infrastructure is fine, just make sure you do know what your doing, and understand your failure modes so when your homelab does go down either for hardware maintenance or due to an issue, you already have a mental picture and plan on how to get at least basic networking back… troubleshooting issues is extra horrible without internet   

Yeah. I'm still on the fence about virtualizing pfsense.

I have a pretty good notion about virtualization in general, but never virtualized or even used a pfsense system. Currently I run a Mikrotik box but the OS its limited in some areas (IPv6, failover and some others, it does those things, just doesn't do them well). I was thinking about a backup plan and I gonna have a pretty straightforward one, leave the Mikrotik box configured but offline, and turn it on if anything breaks Another question is that my plan is to passtrough the whole NIC instead of bridging it with the hypervisor to avoid headaches. I dont know if you have any more tips about virtualizing pfsense it would be extremely helpful.

For somethings like lancache which i'm planning (for data cap questions more than for caching itself) the lower latency will help me a lot. Actually its the reason i'm giving so much RAM to ZFS. It can run fine with less RAM but having lots of ARC help on HDDs. After some research and reflections i've realized that its a better solution to use SSDs that it is faster and less cache, so my plan is to go with 64GB which 16GB DIMMs are WAY cheaper than 32GB DIMMs, less than 1/3 of the price and go with SSDs directly which has lower latency and doesn't need that much cache. One of the problems I currently have is that the seek time for my HDDs even on RAID1 are too big and my connection is not slow (just limited) so I have faster download speeds without something like lancache than with downloading small files. Today my ARC has 12GB reserved if i'm not wrong. Plus SSDs waste far less power, specially on a system that is on 24/7.

About the drives... Well... Lets put it like that. Its not just for the sake of time, i dont think they are reliable anymore. I had them on a couple of seagate USB cases and I got some scrub issues, normally on only one drive but I was suspecting that the USB controller was crapping me out. Its not an ideal solution for ZFS. So i've replaced the USB controller and the problem gone away. That is my suspect but I dont really know if I can trust those drives anymore. Although they dont throw any SMART error even after a long self test and Its not the end of the world loosing the data there. Its the reason I do cloud backup, I don't trust any home network to store important information but I really dont want to use that backup, Iis slow as heck due to the encryption I think its time to retire them. I'll probably leave them there as offline backup just because I woudn't do anything else with them really (in addition to cloud, not replacing it).

[h0m3]

11 hours ago, LIGISTX said:

It’s really not required. It’s a good idea… especially since the point of ZFS is data resiliency at all costs. But….. it’s not needed. There is no scrub or death, your data will be ok (especially if you have a Z2 or greater array, the more drives you have of parity the easier ZFS’s job is at determining who is lying). Plus, Z1 is “dead” with todays drive sizes and costs per drive anyways, no one should be deploying Z1 these days, Z2 really should be the minimum. 

Yeah, that is where I stand. Some bits can be flipped before the ZFS commits or while reading to the ARC cache but... There's nothing in this system that would be unrecoverable in the situation of a couple flipped bits, or maybe the system crashs and you loose the uncommited ZIL, its not crucial in this situation. Thats why for me ECC is more of a bonus, and i do understand that this situation doesn't apply on every scenario, but apply on mine. About Z2 well... I dont think its worth it. If i didn't had any other backup it would make a lot of sense. Theres honestly nothing in that NAS that if I loose the entire storage it would be missed, it would just be an anoyance, and to avoid that specific anoyance I do prefer to have a drive of redundancy but i don't think that two are worth it. Thats actually the reason I didn't replace the HDDs already.

[LIGISTX]

1 hour ago, h0m3 said:

Another question is that my plan is to passtrough the whole NIC

If you do this... only do it this way. Pass the entire NIC through, 1 input for WAN, 1 output for LAN, LAN goes to a switch, and plug your hypervisor into said switch. Don't mess with bridging connections, thats my there.

 

2 hours ago, h0m3 said:

I dont know if you have any more tips about virtualizing pfsense it would be extremely helpful

I wouldn't use your firewall as your first virtualization project... lol. Spin up proxmox and start playing with VM's, but don't try and go straight to virtual networking. I played with VM's under ESXi for a few years, inclduing truenas being virtual with a passed through HBA, and got my feet wet with pfsense running on a separate PC bare metal for about a year. Then I decided to go virtual. The biggest tip is really... make sure you have a good grasp of networking. Ask yourself this "if my homelab is down, which is where my router lives, how do I go about getting back online? If my hypervisor has an issue booting and I need to get into the webUI to fix the issue... how do I do that if my pfsense VM is not booting......?" If you understand how to deal with that, you can probably start virtualizing your router. (The answer is set your laptop or PC to a manual IP address within the pfsesen's physical subnet (if you run vlans.... you need to get your PC onto the physical subnet) and plug into the LAN port directly, then you should be able to hit the pfsense's webUI).

 

2 hours ago, h0m3 said:

After some research and reflections i've realized that its a better solution to use SSDs

I don't think I agree with this. Your networking infrastructure is going to be the limiting factor before reading from a ZFS array... gigabit is 125 MBps, you should be able to read from a harddrive based array much, much faster than 125 MBps. If you want to have fast access to your games, just buy a large SSD and pop it in your gaming PC, that will be MUCH better then a lancache idea... unless you have a bunch of people who live with you and you all want to be able to use the lancache, but even then, it may be a better idea to just get a few SSD's for each PC.

 

2 hours ago, h0m3 said:

I had them on a couple of seagate USB cases

Why not plug them in directly via SATA?

[h0m3]

2 hours ago, LIGISTX said:

I wouldn't use your firewall as your first virtualization project... lol. Spin up proxmox and start playing with VM's, but don't try and go straight to virtual networking

I'm okay with virtualization, my main rig actually runs a VFIO GPU to play games hahaha. Today the only reason that I dont run VMs on that machine is because I dont have enough RAM to do it. So LXD containers it is hahaha. I was really planning to stay on containers if I just run Linux, over LXD different from docker the experience is very similar to a VM, the main difference is that since the kernel is shared you can only run Linux distros and all the kernel modules are on the host, that includes the ZFS and anything else, theres no way to passtrough a NIC for instance because theres nothing to passtrought I could mess with ivshmem but I really dont want to, containers dont need their own memory space. So I actually can save a lot of memory that way. Normally I like to use KVM directly with libvirt so I can manage the hypervisor without a ui over SSH or even directly on the sever TTY if needed but most of the time I do it under virt-manager.

 

2 hours ago, LIGISTX said:

Ask yourself this "if my homelab is down, which is where my router lives, how do I go about getting back online? If my hypervisor has an issue booting and I need to get into the webUI to fix the issue... how do I do that if my pfsense VM is not booting......?"

You have a very good point actually. I didn't tought of that. Its not a problem honestly and your solution is right. But even if I boot my mikrotik back with networking I woudn't be able to put my NAS on the same network to troubleshoot it because it would lead to conflicts (double routing, DHCP and so on). So I would need to actually put a static IP on both machines and access them directly. And at that point there would be zero internet on that machine unless I connect on two separated networks. That would probably require phyisical access to the NAS, which I would tell that its possible but its not the nice office room and its more like the unconfortable attic hahaha.
 

2 hours ago, LIGISTX said:

 

 

 

I don't think I agree with this. Your networking infrastructure is going to be the limiting factor before reading from a ZFS array... gigabit is 125 MBps, you should be able to read from a harddrive based array much, much faster than 125 MBps. If you want to have fast access to your games, just buy a large SSD and pop it in your gaming PC, that will be MUCH better then a lancache idea... unless you have a bunch of people who live with you and you all want to be able to use the lancache, but even then, it may be a better idea to just get a few SSD's for each PC.

Actually my problem is more related to latency than speed, downloading large sequencial data is fine and I hit the 125MB/s no problem, but when something hits on random read the HDDs slow down, as expected. And since I have a pretty good fiber (two actually, they just are bandwidth limited) they can actually download random files way faster than my HDD can respond to them. I examplified with lancache because its a example of actuall small blocks of file cached trought squid but anything cache related tends to be chopped in small blocks.

And believe or not. Unless you go way high on space here a WD Red of lets say 1TB is the same price as a 1TB SATA SSD with DRAM. The price only starts to change once you pass the 2TB mark, 4TB NAS HDD are cheaper than 4TB SSDs but I dont wanna go that high on density. I dont need that much space.
 

2 hours ago, LIGISTX said:

Why not plug them in directly via SATA?

My current NAS Dell box is the smaller size of those SFF. It was a repurposed machine and I didn't have the budget for a proper NAS setup. The machine only supports one SATA 2.5 inch and one NVMe drive. Theres no extra SATA ports and no PCI-Express to add a controller. My only option was USB. And fairly enough it does work, has its issues but it does work hahaha.