FreeNAS + Pfsense on a single machine to handle gigabit fiber internet.

[lowstrife]

Starting to run into bottlenecks\headaches on my "one computer to rule them all". I recently upgraded to 3rd gen ryzen, so I have some spare hardware sitting around to address these problems. 

 

My current solution + it's problems: 

 

Gigabit fiber + Cable backup operating in failover

Edgerouter Lite

Ryzen 3900x & 64GB memory

2x4TB and 2x8TB discs manually mirrored once a week. 

2x1TB nvme SSD for boot & scratch disc

 

Problem #1: The Edgerouter can't handle the loads I'm putting on it. It can do gigabit to a single host (Speedtest.net), but in the real world it falls apart. This screenshot is ~400mbps overall network load, yet my connection can handle symmetrical gigabit in both directions co-currently. Yes, I am running with hardware offloading enabled. That's why I bought this router, but it's still not enough.

 

 

Problem #2: Local internet is only gigabit, and if I'm building a NAS, I'm going to need a 10 gig link to handle the media files I work with.

 

Problem #3: My current backup solution is manually mirroring the data, and it sucks. And it leaves gaps. And it's space in-efficient. 

 

Problem #4: My current storage is filling up. I'm sitting at about 80% capacity on my spinning discs. 

 

Problem #5: I now have a Ryzen 1700, motherboard, SSD and power supply just sitting around. Oddball parts to sell, especially in this economy. Would be worth far more to me to put them to work rather than selling for 20 cents on the dollar. 

 

____________________________________________

 

My proposed solution:

 

 

I would build a new machine out of these parts. Run a hypervisor on a base OS to support FreeNAS and pfsense in their own VM's with hardware passthrough so FreeNAS can get the hardware access it needs to the discs. Get some add-in PCI-e network cards so the box can have the necessary ethernet ports, including 10 gig to my main machine. I don't want to mix\match the 1gb lines with the onboard NIC so I will be getting a 4-port NIC. 

 

Questions that I haven't been able to answer in my research for this setup:

 

1) For security reasons, I've seen that you want your fileserver as far away as possible from the internet. But this fileserver will only be hosting media, nothing sensitive. And since they are on separate VM's, they will be "seperated". People say not to run pfsense ontop of Freenas, but what about running them both side-by-side ontop of Ubuntu or some other distro? Will network & firewall configuration play nicely? 

2) Is it going to be a problem running pfsense and freeNAS within their own VM's? The hardware is powerful enough to do it, but I'm worried about stability & disaster recovery. 

3) How will disc partitioning work for running these layered systems? I have the 250GB SSD, will it be possible to partition part of that to the operating systems, and then leave the rest for a SSD cache to the fileserver? 3x 16GB partitions with the rest dedicated to the FreeNAS cache. 

4) What "root" operating system should I use to host all of this, and what hypervisors would be recommended to host these two instances. 

5) I'm looking to run RaidZ-2 for double redundancy. Using 4 new 8TB discs to start the array, migrating my data onto it, then expanding the array using my existing two 8TB discs.

6) If my whole network is 1gb, is pfsense able to identify the 10 gig link to my workstation and allocate that accordingly so that I can have higher speeds to the fileserver?

 

Or... Do you guys think this a stupid idea and I should just build a dedicated box for both. 

 

Thanks for the input everyone. 

[Windows7ge]

1. If you have setup a bridged network adapter on the host the VMs will appear to the network as separate machines. They shouldn't interfere with one another if that is your question.
2. a Router OS is one of those things I don't recommend virtualizing. You will have to take the server down for maintenance from time to time and things can go wrong. If this happens it means you'll lose your router which means your whole home will lose internet. You should really use a dedicated box for pfSense.
3. If you use a disk for the host it cannot be passed through to a VM for the purposes of a cache disk. It's either one or the other. Can't have both unless you create a 4th virtual disk and give that to the VM for caching but it'd come with unnecessary overhead.
4. I like PROXMOX but there's also ESXi, HyperV, QEMU/KVM, VirtualBox. Some are more recommendable over others.
5. ZFS pools can only be expanded by appending vdevs. Each vdev requires it's own redundancy. If you create an initial raidz2 with four drives (might as well use RAID10 for more IOPS then) then when you append the next two drives you'd have to use a mirror. Reason is the data would be striped across the vdevs. If you didn't add redundancy to the 2 new disks and one failed all the data in the pool would be lost.
6. When you connect to a local server the router is irrelevant. With a VM server and a bridged network adapter it creates a virtual switch. This will contain a forwarding table which operates at Layer 2. If you specify the IP of the 10Gbit interface it will connect you over that network link. Your bigger concern should be if Windows will use the faster of the two links when using hostnames as it doesn't always.

Dedicated box for the router is my recommendation. I wouldn't run FreeNAS in a VM either. Use something like PROXMOX if you want ZFS and a good hypervisor or alternatively Debian(ubuntu/mint/popos/lubuntu/etc)+QEMU/KVM+Virt-manager+zfs-utils.

[Paroxy]

1) If they both running in their own VMs and you setup the virtual networking properly this should be fine security wise.

 

2) You can take regular snapshot images of your VMs and back them up. 

 

3) You assign a chunk of physical storage space to be a virtual hard drive for the VM.  Inside the VM it works just like a regular hard drive.

 

4) UnRAID would be my first recommendation.  Proxmox is another option, it's more flexible but not beginner friendly.  Windows 10 Pro running Hyper-V is another option.

 

5) Get a proper hardware SAS RAID card. Like something made by LSI, Intel, Dell, or HP.  For my file server I went with a used one off eBay and it has worked great.

 

6) It should be fine but I doubt your RAID array will be able to transfer that fast.

[lowstrife]

8 hours ago, Windows7ge said:

1. a Router OS is one of those things I don't recommend virtualizing. You will have to take the server down for maintenance from time to time and things can go wrong. If this happens it means you'll lose your router which means your whole home will lose internet. You should really use a dedicated box for pfSense.
2. ZFS pools can only be expanded by appending vdevs. Each vdev requires it's own redundancy. If you create an initial raidz2 with four drives (might as well use RAID10 for more IOPS then) then when you append the next two drives you'd have to use a mirror. Reason is the data would be striped across the vdevs. If you didn't add redundancy to the 2 new disks and one failed all the data in the pool would be lost.

Dedicated box for the router is my recommendation. I wouldn't run FreeNAS in a VM either. Use something like PROXMOX if you want ZFS and a good hypervisor or alternatively Debian(ubuntu/mint/popos/lubuntu/etc)+QEMU/KVM+Virt-manager+zfs-utils.

 

Alright you gave a ton of insight, thank you. I just have a couple followups that I edited down:

 

1) This sounds like a pretty big no. I've seen... too many people recommending that you shouldn't visualize these things now. Especially when we're talking about hardware passthrough, network bridging... That does sound really messy. 

 

2) So you can't add more drives to an existing vdev (or at least RaidZ-1 or Raid Z-2). Once you set that up, it's locked in. And the only way to expand the pool is to create an entirely new vdev. Got it. This sounds like I am leaning toward UnRAID at this point because it's easier to migrate and add additional storage in a... cleaner way.

 

I just have one final question at this point:

 

Dedicated box for the router. Where should I go here. Most people on most normal connections can get away with a cheap-o Atom or Pentium legacy processor in a 10 year old Optiplex. However, I am going to need something quite high performance - as I explained above - but I've been having a really difficult time tracking down the best way to do this without just throwing expensive hardware at the problem. My initial conclusion isn't far from my original plan:

 

Dedicated box running pfsense with RJ45 add-in card with 4 1GB ports

 

Does it make sense to run a router basically with PCI-e add-in cards? And how well would this setup scale if I were to add 10GB add-in cards in the future so that I can have a workstation <--> fileserver link be at the full 10gb speed. The pre-configured hardware from them has some fairly low routing bandwidth once you start getting into 10gb links and you need a pretty beefy quad core chip to properly saturate a 10gb link. 

 

 

7 hours ago, Paroxy said:

4) UnRAID would be my first recommendation.  Proxmox is another option, it's more flexible but not beginner friendly.  Windows 10 Pro running Hyper-V is another option.

 

5) Get a proper hardware SAS RAID card. Like something made by LSI, Intel, Dell, or HP.  For my file server I went with a used one off eBay and it has worked great.

 

6) It should be fine but I doubt your RAID array will be able to transfer that fast.

Also thanks for your input. 

 

4) As seen above, I think I am leaning more toward UnRAID at this point. 

 

5) I can understand using one if you need a HBA because you've run out of motherboard slots, but why would you need a hardware RAID card on a software UnRAID system?

 

6) I am using media files, so it will be max sequential reads. A single platter can saturate a 1 gig link most of the time unless it's reading data on the inside of the disc. 5-6 disc benchmarks I've seen usually land somewhere in the 250-500 MB\s range for sequential read\write speeds. Especially since I have local NVME storage on the other end of this fileserver.

 

But I think I will be going the dual-system approach. Existing Ryzen 1700 hardware for the fileserver, new hardware for the pfsense router. I think you guys walked me back from the edge of trying to combine them onto one system.

[Paroxy]

Sweet, sounds like a solid plan.

 

I started with ProxMox before switching to UnRAID, and ProxMox does not do software RAID.  I stuck with the card when I later switched to UnRAID because I assume it lightens the load on the CPU, and the motherboard I was using had a mixture of SATA port speeds, plus having more SATA ports is nice when UnRAID supports an unlimited number of drives.

[Windows7ge]

1. This entirely depends on what is is you're looking to do.
   1. databases
   2. workstations
   3. website hosting
   4. stock trading
   5. file servers
      1. For many server applications it's very appropriate to use virtualization in order to save on space/power but your router isn't one of those things you want to virtualize. If the server goes offline (and it will) then everyone is offline. 
2. ZFS is a more enterprizy File System. It's not friendly for all home user applications such as mixing drives of different capacities or single disk addition. You can add storage to an existing pool but as you said you cannot expand an existing vdev. Each vdev you append needs to contain it's own redundancy. This is why if you had, let's say 6 drives in a raidz2 you'd want to append a vdev of another 6x drives in another raidz2. No adding 1 disk at a time.
   1. It's a great File System but it has it's gotchas.
3. As you enter the 1Gbit and especially the 10Gbit territory pfSense has it's own bottlenecks where you may not see the full performance of your internet connection. pfSense will run on just about any modern hardware and act as a router far more powerful than any off the shelf solution but it has it's drawbacks. If you're worried about the hardware having enough power a platform/cpu with lesser cores but higher core clock would yield the best results. Don't go crazy with it but this general principle would be best for pfSense. Something like LGA1151. LGA1150 would probably still do you just fine. Whatever is easier/cheaper to source.

[2FA]

I had my lab router virtualized for a time, was fine except for losing internet access when the server restarted which I wouldn't say was frequent but was often enough it was annoying. You can get a ProtectCLI, Qotom, or Fitlet2 box decently priced which would give a lot more performance for the price than Netgate's offerings. I personally have a PCEngines APU4 running VyOS but that can only do gigabit with Linux-based routers and not BSD-based ones like pfsense because of software inefficiency.

[lowstrife]

Alright, thanks so much for the suggestions everyone. I have one final question then:

 

I do need to have a priority on the gigabit internet connection being stable and high performance. What router solutions should I be looking for that can handle this? Actual, real,  gigabit duplex. Needs to be able to handle PPPOE. Obviously you can spend the $800,000 on the Cisco solutions, is there anything out there that is more achievable on the prosumer or light commercial side of the spectrum? 

 

I think that using an actual dedicated SFP+ switch to handle 10 gig internal LAN would be the best, and leaving the dedicated router to do dedicated router things inbetween the fileserver and the workstation. I will be using the Ryzen CPU & legacy hardware I have solely for the fileserver\VMhost at this point I believe.