Need advice for a server setup to act as a router as well as NAS

[k4har]

I was thinking of using PFSense to act as a router and do all the networking, please give your opinion if you think there is a better alternative. Unsure about what to do in terms of the NAS/Media server.

 

Current setup:

Asrock X370 Gaming K4 motherboard

Ryzen 2200G CPU

8GB RAM

250GB SSD

4TB HDD

3x  4 Port gigabit NICs (overkill i know but i got them cheap)

 

Network will consist of 3 BT Whole Hub Mesh WiFi connected via ethernet, these will cover the WiFi throughout the house.

There will be a total of 7 ethernet cables running from the server throughout the house, 8 if you include the one from the modem.

 

Should i run a cable from the modem (Virgin Media Superhub 3 in modem mode) to the motherboards NIC or straight to one other PCI NICs?

 

The server will also be connected to a TV in the living room to act as home theatre system.

 

[Windows7ge]

In my opinion these are two things that shouldn't be mixed. Can you? Yes. Should you? I don't think so.

 

It's routine to take servers offline for maintenance, upgrades, etc and that would take out your router too which means no internet to the rest of your home/office/etc until the NAS is back online. What if the whole NAS OS goes way off kilter or you experience catastrophic hardware failure? You'll be without internet until you reset your modem or find a temporary router.

[Spotty]

5 minutes ago, k4har said:

Unsure about what to do in terms of the NAS/Media server.

The way I would do it would be to set the machine you have as a NAS with FreeNAS & use Plex plugin as your media server. Probably not worth the hassle of running a virtual machine to run pfsense to handle networking on the same machine as the NAS. 16 port gigabit switches start at about $50-$100 (and go in to the thousands), but for a media server you should be right with the basic stuff. Use one of these for networking instead.
Connect all the ethernet cables (PCs, TVs, Wifi boosters, Modem, and NAS) to the network switch.

You're probably going to want to consider more hard drives and more storage space as you will be surprised how quickly 4TB will fill up, but that's a whole other rabbit hole.
 

[k4har]

6 minutes ago, Windows7ge said:

In my opinion these are two things that shouldn't be mixed. Can you? Yes. Should you? I don't think so.

 

It's routine to take servers offline for maintenance, upgrades, etc and that would take out your router too which means no internet to the rest of your home/office/etc until the NAS is back online. What if the whole NAS OS goes way off kilter or you experience catastrophic hardware failure? You'll be without internet until you reset your modem or find a temporary router.

2 minutes ago, Spotty said:

The way I would do it would be to set the machine you have as a NAS with FreeNAS & use Plex plugin as your media server. Probably not worth the hassle of running a virtual machine to run pfsense to handle networking on the same machine as the NAS. 16 port gigabit switches start at about $50-$100 (and go in to the thousands), but for a media server you should be right with the basic stuff. Use one of these for networking instead.
Connect all the ethernet cables (PCs, TVs, Wifi boosters, Modem, and NAS) to the network switch.

You're probably going to want to consider more hard drives and more storage space as you will be surprised how quickly 4TB will fill up, but that's a whole other rabbit hole.
 

Will running Pfsense and maybe FreeNAS in a VM hurt Pfsense networking or R/W performance of FreeNAS. Im currently waiting for my 12TB drives to come back from RMA as BOTH were DOA.

6 minutes ago, RandomGuy13 said:

You can do either, Your motherboard has a decent intel nic so should work absolutely great.

 

As for 4port NIC recommendations, Anything intel would work great, I am using a PRO PT1000 2 port card like this one, but intel also do a couple 4 port versions like This IBM one or This one.

I realy like the Pro PT cards because they are super cheap on ebay.

Other cheaper ones that you might find on amazon are not as fast, you won't realy get the full gigabit, and they might not support features like jumbo packets, or be properly supported by PFsense.

The NICs are IBM Intel I340-T4 haven't really looked at specs just picked them up because they were super cheap on gumtree.

 

[Spotty]

1 minute ago, k4har said:

Will running Pfsense and maybe FreeNAS in a VM hurt Pfsense networking or R/W performance of FreeNAS.

On a 4 core CPU with only 8GB of RAM? There likely would be a noticeable impact.

Are you experienced enough with virtual machines, FreeNAS, and pfsense that every time someone in your family goes "AHHH the internet isn't working properly! FIX IT!" you will be capable of troubleshooting the issues yourself and getting everything back online?
As @Windows7ge stated, if something goes wrong and you need to take the system offline for maintenance, troubleshooting, or hardware replacements/upgrades, you will lose your NAS and your networking.

[Mikensan]

FreeNAS as a VM requires the ability to give it direct access to the HDD controller. You could instead use NAS4Free + UFS in a VM however and it would be fine. 

 

If you're looking for security from pfSense then it should not run parallel to anything else. However isolated inside a VM and using hardware passthrough, give it direct control/access to ports on a NIC, it is reasonably safe. This would be a big no-no in a corporate environment but for home use I don't see why not. 

 

You have a few big name hypervisors to chose from... ESXi, KVM (Proxmox to name one), or Xenserver. Personally I like ESXi, but different folks different strokes.

 

ESXi is going to want about 1gb for itself, pfSense for a normal home should have about 2gb if you plan to torrent, leaving you with 4-5gb for your NAS.. Which is plenty for general home use.

 

It just sucks when you have to restart the entire server for whatever reason, that you lose network connectivity.

 

In the future I'd save about ~$250 and maybe either get a m-ITX solution or R210 ii to host pfsense on.

http://a.co/ie2nqaw 

^ That's only $150, and it's the officially supported hardware for pfSense.

[Windows7ge]

59 minutes ago, k4har said:

Will running Pfsense and maybe FreeNAS in a VM hurt Pfsense networking or R/W performance of FreeNAS

I don't have enough experience with pfSense to answer that question but depending on what virtualization software you use it will add different amounts of latency to system requests. Now FreeNAS likes direct access to the drives which means no hardware RAID controllers. Use the ZFS software RAID. Some VM software such as ESXi & UnRAID support hardware pass-through but again I have no experience with this so I can't say how this will influence FreeNAS's ability to control the drives.

 

1 hour ago, k4har said:

Im currently waiting for my 12TB drives to come back from RMA as BOTH were DOA.

...I have no words to describe how surprising that is. Either the delivery guy slammed it into the ground like a football (American) touchdown or maybe they weren't dead just someone thought they were. Do drives show up DOA? Yes. Do TWO drives show up DOA? WAAAAY less likely unless some serious crap happened in shipping. Was the box they showed up in severely damaged?

[k4har]

2 hours ago, Spotty said:

On a 4 core CPU with only 8GB of RAM? There likely would be a noticeable impact.

Are you experienced enough with virtual machines, FreeNAS, and pfsense that every time someone in your family goes "AHHH the internet isn't working properly! FIX IT!" you will be capable of troubleshooting the issues yourself and getting everything back online?
As @Windows7ge stated, if something goes wrong and you need to take the system offline for maintenance, troubleshooting, or hardware replacements/upgrades, you will lose your NAS and your networking.

If need be i could switch to a Ryzen 1700 or install another 8GB RAM stick. I currently have 2 connections at home, one is cable broadband and the other is standard adsl broadband, both are permanently on just incase one goes down. I was looking into getting a load balancing router so it auto switches if one goes down, but the cable connection hasn't dropped a connection since it was installed 2 years ago, only for firmware updates.  I work for an ISP so know how basic networking works and i have gone into the more advanced networking at bit. Starting to work towards my CCNA certification next month. The only time this system will be upgraded is when the hard drives from Seagate arrive. I might pop in the 1700 if i need more processing power or more RAM.

1 hour ago, Mikensan said:

FreeNAS as a VM requires the ability to give it direct access to the HDD controller. You could instead use NAS4Free + UFS in a VM however and it would be fine. 

 

If you're looking for security from pfSense then it should not run parallel to anything else. However isolated inside a VM and using hardware passthrough, give it direct control/access to ports on a NIC, it is reasonably safe. This would be a big no-no in a corporate environment but for home use I don't see why not. 

 

You have a few big name hypervisors to chose from... ESXi, KVM (Proxmox to name one), or Xenserver. Personally I like ESXi, but different folks different strokes.

 

ESXi is going to want about 1gb for itself, pfSense for a normal home should have about 2gb if you plan to torrent, leaving you with 4-5gb for your NAS.. Which is plenty for general home use.

 

It just sucks when you have to restart the entire server for whatever reason, that you lose network connectivity.

 

In the future I'd save about ~$250 and maybe either get a m-ITX solution or R210 ii to host pfsense on.

http://a.co/ie2nqaw 

^ That's only $150, and it's the officially supported hardware for pfSense.

I've been looking into ESXi and seems like the best bet but haven't done much research on it as of yet. I am correct in saying that ESXi need direct access to the whole drive and i can't partition the SSD and put Pfsense on one partition and say FreeNAS on the other? I haven't played with VMs since i needed to know how to use Linux properly, a long time ago.

1 hour ago, RandomGuy13 said:

They should work great, anything intel works great in my experience nothing else realy compares especially not the cheap off brand ones.

Thats good to hear.

1 hour ago, Windows7ge said:

I don't have enough experience with pfSense to answer that question but depending on what virtualization software you use it will add different amounts of latency to system requests. Now FreeNAS likes direct access to the drives which means no hardware RAID controllers. Use the ZFS software RAID. Some VM software such as ESXi & UnRAID support hardware pass-through but again I have no experience with this so I can't say how this will influence FreeNAS's ability to control the drives.

 

...I have no words to describe how surprising that is. Either the delivery guy slammed it into the ground like a football (American) touchdown or maybe they weren't dead just someone thought they were. Do drives show up DOA? Yes. Do TWO drives show up DOA? WAAAAY less likely unless some serious crap happened in shipping. Was the box they showed up in severely damaged?

Box? If only they had packaging. Ordered from amazon prime now, they only came in an anti static bag which was in a brown paper bag. No padding whatsoever. I think the R/W heads were misaligned because both were making a noise as if metal is hitting metal, like a grinding noise.

[Windows7ge]

9 minutes ago, k4har said:

Box? If only they had packaging. Ordered from amazon prime now, they only came in an anti static bag which was in a brown paper bag. No padding whatsoever. I think the R/W heads were misaligned because both were making a noise as if metal is hitting metal, like a grinding noise

That sounds like a high probability of shipping damage. When I get WD drives from Newegg then come inside a anti static bag with plastic braces around both ends of the drive which suspended the drive in the air so as to adsorb all shock from impact. This is put inside a box and then the box itself is put inside another box with padding. You'd have to try HARD to hurt it. But yeah, anti static bag put in a paper bag. Dropping that once onto concrete could damage it.

[Mikensan]

Quote

I've been looking into ESXi and seems like the best bet but haven't done much research on it as of yet. I am correct in saying that ESXi need direct access to the whole drive and i can't partition the SSD and put Pfsense on one partition and say FreeNAS on the other? I haven't played with VMs since i needed to know how to use Linux properly, a long time ago.

ESXi can work off a USB flash drive, and you can then use the whole SSD for virtual machines. Virtual machines use a file called "VHD" or "VMDK" which are virtual hard drives, and that way the VMs would have storage. The issue isn't ESXi it is FreeNAS. You can just use NAS4Free, it supports UFS which just doesn't care. FreeNAS doesn't just want the whole drive, but the controller and subsequently all disks connected to it. 

 

Personally I'd spend $150 on the pfsense box at some point, and dedicate the entire machine to a NAS solution. However it really depends on what you want to do.

[k4har]

7 minutes ago, Windows7ge said:

That sounds like a high probability of shipping damage. When I get WD drives from Newegg then come inside a anti static bag with plastic braces around both ends of the drive which suspended the drive in the air so as to adsorb all shock from impact. This is put inside a box and then the box itself is put inside another box with padding. You'd have to try HARD to hurt it. But yeah, anti static bag put in a paper bag. Dropping that once onto concrete could damage it.

The bag didn't even have the usual sticker that seals it, no shipping label, just a brown bag. I did get 6 months free amazon prime, as they sent me an Athlon 64 X2 in a ryzen 2200g box, the box had 4 chipset heatsinks from like the 90s. The CPU itself even had a clear sticker on it to make it look like and be labeled like a 2200G. Oh and it had 4 bent pins. GG amazon. 

2 minutes ago, Mikensan said:

ESXi can work off a USB flash drive, and you can then use the whole SSD for virtual machines. Virtual machines use a file called "VHD" or "VMDK" which are virtual hard drives, and that way the VMs would have storage. The issue isn't ESXi it is FreeNAS. You can just use NAS4Free, it supports UFS which just doesn't care. FreeNAS doesn't just want the whole drive, but the controller and subsequently all disks connected to it. 

 

Personally I'd spend $150 on the pfsense box at some point, and dedicate the entire machine to a NAS solution. However it really depends on what you want to do.

Ok i understand now. I've just checked my TVs which will be the only devices accessing the media, which is mainly videos, they can play all the videos natively, so i don't even need something like plex to transcode. All i need is stream the file as is to the TVs. Looks like FreeNAS is off the table because i only have the onboard controller and don't feel like investing into a proper RAID or SATA controller.

[jde3]

FreeBSD is the root project of both PFSense and FreeNAS so ovis it can do the jobs of both.

 

Is it a good idea? Sure if you know what your doing. Usually for an edge router I like OpenBSD but there isn't a real good reason FreeBSD can't do it. There is also plenty of documentation out there on how to do this... who knows maybe I'll write a guide up here, for a firewall with jailed services.. dunno.

[k4har]

28 minutes ago, jde3 said:

FreeBSD is the root project of both PFSense and FreeNAS so ovis it can do the jobs of both.

 

Is it a good idea? Sure if you know what your doing. Usually for an edge router I like OpenBSD but there isn't a real good reason FreeBSD can't do it. There is also plenty of documentation out there on how to do this... who knows maybe I'll write a guide up here, for a firewall with jailed services.. dunno.

I haven't looked into FreeBSD, so it can have all the functions of Pfsense and FreeNAS as one OS?

[jde3]

Yeah.. all PFSense and FreeNAS are.. essentially are web frontends for FreeBSD. (NAS4Free too)

 

Pretty much the high level overview of what you need to do is.

 

Install it.

Configure your ZFS pools.

Configure NFS, Samba, Syncthing.. whatever.

Turn on ip forwarding and PF

Configure PF

Add a jail manager like iocage and install plex, emby, nextcloud, openvpn.. whatever.

 

good to go.

[k4har]

And would a Ryzen 2200G and 8GB RAM suffice for what i need?

[jde3]

I'm unsure if it supports the graphics driver with anything more than vesa mode. but.. ya.. should work no problem.

 

How much storage?

[jde3]

Actually I think you need to disable the temp sensor on Ryzen on 11.1-RELEASE because it throws errors.. not sure if thats been patched or not.

[k4har]

Ill download and install it see what the driver loads up, if not i do have a few old fanless GPUs laying around somewhere. Thanks for that i'll disable it if i have any problems. Currently i have a 250 SSD and 4TB HDD, but will be taking out the 4TB for 2x12TB drives once they come back to me.

[jde3]

It dosen't really do anything to the board or anything it's just a glitch in how it's read by FreeBSD.. as I remember.. (idk I shut it off once right after ryzen came out and never looked at it again.) If your only running in console mode the graphics should be fine.

 

You'll want to plan out your storage pool and set your dataset options up before you add the data.

Jails will use their own zfs dataset and they are like mini isolated VM's. Ezjail or iocage are popular jail managers. (text based tho)

PF is a little harder to get into.. it's also not the default firewall in FreeBSD (thats ipfw, same one used by apple)

There are guides out there and you can write highly dynamic and short firewall that should be able to take care of it.. but do test it with nmap.

There is a package manager called "pkg" the new version of this uses a similar syntax to apt so it's pretty easy to use.

 

In most cases the commands are very similar to Linux but there is some small differences (usually in the switches for the commands) if you get stumped do man "command", or man "config.file" - FreeBSD has much better man pages than most Linux distros.

 

It might take you some time to get all this worked out and learn it.. The handbook on FreeBSD's site is top notch.. it's one of the better OS manuals/wiki's out there. If your ever in doubt and the man command can't help you, turn there for help.

[k4har]

Great stuff, i'll let you guys know if i need any help.

[jde3]

One other non-linux'ey thing it does.. it places all the software you add from pkg or ports into /usr/local .. the rest of the system is what they call "base" so it separates any "add on software" to /usr/local - this is weird at first if you know linux but it's nice because you can keep clean separation on what you base OS is and what was added. So config files from pkg software will show up in /usr/local/etc

[k4har]

Ok thanks

[ltguy]

I would not use the pfsense box to do the switching to all 7 ethernet cables. Software switching is much slower than an actual switch. If you want to use more than 1 ethernet cable in a link agg group to run to a managed switch fine, but let a hardware switch do the switching.

[k4har]

On 27/04/2018 at 6:29 PM, ltguy said:

I would not use the pfsense box to do the switching to all 7 ethernet cables. Software switching is much slower than an actual switch. If you want to use more than 1 ethernet cable in a link agg group to run to a managed switch fine, but let a hardware switch do the switching.

Thanks didn't know that, waiting for my RMA on my motherboard as the second PCI express slot doesn't work. I haven't had a chance to try FreeBSD.