Sanity Check Needed: Dropping $3000 for pfsense box and home server.

[bloodthirster]

14 minutes ago, dalekphalm said:

Not bad - but you can find newer generation Xeons for the same or cheaper. Example, you can find Xeon E3-1220 V3's (That's the Haswell version) for in the $50 per socket ballpark. 2x 1220 V3's would smoke the 6380's in most scenarios.

True - but your CPU's are also very power hungry. If he doesn't need much CPU power, he's better off doing a low power build anyway.

Depends on the specific builds. As I've shown above, if I can get Xeons for the same price as your old opterons, it's a much better perf/dollar ratio than what you got.

AMD's older CPU's also had security flaws, so that's a wash at best.

 

I have no problem with AMD - and their Zen based server CPU's are kickass (Threadripper and and EPYC). But their Bulldozer derived architecture (including Piledriver, yours), was pretty terrible to begin with - it was an unfortunate mistake that AMD made, and it hurt them for a long time.

Intel is definitely a valid option, and I've shown you can easily do very good using both Intel or AMD.

Meltdown has large perf hits.  Also, those E3-1220 V3s are SMT which also has security flaws.  Those can't be run in a dual socket MB either.   Intel has more perf killing security flaws than AMD.  That you can't argue.

 

Yes, Bulldozer (and refreshes) aren't perfect but they aren't bad either.

 

14 minutes ago, dalekphalm said:

They can be, yes. But it entirely depends on what the OP wants to use said ports for. The SATA ports remain usable for the Hypervisor directly. I don't think it's as big a concern as you're implying. Especially since good used controllers are pretty cheap on eBay.

Yes... yes you can. In ZFS for example, you create your overall array, then you can easily portion some out to iSCSI, leave the rest for SMB/Samba or NFS, etc.

Wait, there are controllers which can pass SATA ports to a VM without overhead?  Because to my knowledge, you have to pass the whole controller over to the VM.  Other wise you're getting world switches anytime you want to access the HDDs/SSDs.

 

Still messy.

14 minutes ago, dalekphalm said:

And while iSCSI over a VM is overkill... so is this entire setup? If we wanted the non-overkill option, it'd be to buy an off the shelf QNAP or Synology and call it a day.

Except that other OS's that are more storage focused tend to have shit VM support - such as FreeNAS. Sure you can run VM's on FreeNAS, but it's not a great experience.

 

Running my FreeNAS VM on ESXi was the best decision I ever made with my home server. It runs flawlessly, with bare metal performance, and I don't have to deal with shit VM support.

It just doesn't make any sense...

 

Most Linux distros have ZFS packages (Debian, Ubuntu, Gentoo I know all do) and KVM works pretty damn well.   You make it sound like you either have to have a complex set up or crap VM support.

 

14 minutes ago, dalekphalm said:

Granted, ESXi is not the only choice. If your concern is storage support, then use ProxMox instead - it's linux based, has ZFS support out of the box, and is an excellent Type-1 Hypervisor.

Linux isn't a Type 1.  Xen is... but Linux and KVM is NOT.    It may ACT like a type 1, but Linux will never be a type 1.   If you want a true Type 1,  use Xen (especially after they get PVH dom0s working).   Xen is a proper type 1 which works well.  Linux/KVM works well too but is a type 2.

 

14 minutes ago, dalekphalm said:

I don't think virtualizing a router is a terribly good idea to begin with, but in the vast majority of home user scenarios, performance won't be an issue. It could be a problem - but again, depends on the scenario.

It can be useful, and it can also be a learning experience.

 

How much the OP wants to virtualize is entirely up to him. He may decide that it's not worth the trouble, and do a different setup. Or he may decide to learn the setup.

 

We've given him advice. If he has questions, we'll address those if they come up.

 

I'm not saying your suggestions are bad - they have merit, but you're dismissing things that are pretty standard procedure as if they're scary unheard of processes.

People are suggesting things which don't make sense (virtualizing a NAS) when there are better  and more simple solutions.  It's like suggestion gentoo to someone who's has a complaint about Win10.    If the OP knows what he's getting into, go for it, but that doesn't mean it's the best course of action.

 

Standard procedure isn't to put high IO basic functions in a VM.   They aren't scary, it's just unneeded and makes things more complex than they have to be.  K.I.S.S.

[GlorifiedPaperShuffler]

Back to the hardware a little...

I still don't know WHY I'd buy a Xeon Scalable Silver for the purposes of a home server. I can certainly afford it, but at this moment it feels like I'm buying a Lambo as a grocery getter.

 

I'm not saying I'd want a Corolla as a grocery getter either, even though it is fit for purpose. Perhaps something in the Mercedes Benz range?

What would be a processor equivalent to the Mercedes Benz range?

Edited March 23, 2019 by DaAznKnight

[dalekphalm]

8 hours ago, DaAznKnight said:

Back to the hardware a little...

I still don't know WHY I'd buy a Xeon Scalable Silver for the purposes of a home server. I can certainly afford it, but at this moment it feels like I'm buying a Lambo as a grocery getter.

 

I'm not saying I'd want a Corolla as a grocery getter either, even though it is fit for purpose. Perhaps something in the Mercedes Benz range?

What would be a processor equivalent to the Mercedes Benz range?

I'd skip the Silver Edition entirely, and just go with a bog-standard (as the brits would say) Xeon.

Do you want one socket (One CPU) or 2? If one, just grab an E3 Xeon, with however many cores you can (or want) to afford. If you need dual socket (two CPU's), grab an E5 Xeon and a motherboard to match.

[GlorifiedPaperShuffler]

17 hours ago, dalekphalm said:

I'd skip the Silver Edition entirely, and just go with a bog-standard (as the brits would say) Xeon.

Do you want one socket (One CPU) or 2? If one, just grab an E3 Xeon, with however many cores you can (or want) to afford. If you need dual socket (two CPU's), grab an E5 Xeon and a motherboard to match.

Hrm, I was thinking of going 1 CPU for now, with a dual socket motherboard. Just leave one socket unpopulated, room for upgrades later.

Does that work? Again, never played with dual socket stuff before, just like I've never played in the Xeon range.

 

I changed the specs around, to use Xeon E5s instead of Silver Scalable. E5 was chosen because..., that seems to be what everybody on YouTube is using.

Also decided that maybe it's worth it to keep the pfsense box separate, because I swear my wife and daughter are going to murder me if they can't access the Internet for hours at a time whenever I wanna tinker.

 

Primary source of confusion now is if the HBA Card is the "correct" card for this. It's been wildly confusing, with some talking about RAID, and others saying you need to flash it, etc. etc.

[dalekphalm]

14 hours ago, DaAznKnight said:

Hrm, I was thinking of going 1 CPU for now, with a dual socket motherboard. Just leave one socket unpopulated, room for upgrades later.

Does that work? Again, never played with dual socket stuff before, just like I've never played in the Xeon range.

Yes that works. Keep in mind, correct memory slot usage is important in a Dual Socket motherboard.

 

If you start with a single CPU, make sure to populate the RAM slots connected to that CPU only (generally the RAM slots are on either side of the CPU's, so it's pretty obvious which CPU controls which RAM - consult the manual if unclear, or ask for advice).

 

Then, when you decide to upgrade to a 2nd CPU, you need to buy RAM at the same time (or re-arrange your existing RAM) to populate RAM slots on the 2nd CPU's side.

14 hours ago, DaAznKnight said:

I changed the specs around, to use Xeon E5s instead of Silver Scalable. E5 was chosen because..., that seems to be what everybody on YouTube is using.

Good call. You can buy 2x (and have some change left over) of the E5's you've chosen for the cost of a single Silver edition. Is the Silver Scalable CPU better? Yes. Is it twice as good? Helllllll no. We're talking a core difference of 8 for the E5 vs 12 on the Silver. 2x E5's are cheaper than one Silver 4116, and combined more powerful.

14 hours ago, DaAznKnight said:

Also decided that maybe it's worth it to keep the pfsense box separate, because I swear my wife and daughter are going to murder me if they can't access the Internet for hours at a time whenever I wanna tinker.

Good call to not virtualize it. However, I'd still recommend against doing a full blown pfSense build - as others have pointed out, grabbing a hardware pfSense router is a better idea. You save a lot on the hardware cost, and you can still tinker in full at the software level.

14 hours ago, DaAznKnight said:

Primary source of confusion now is if the HBA Card is the "correct" card for this. It's been wildly confusing, with some talking about RAID, and others saying you need to flash it, etc. etc.

The LSI 9300-8i is an HBA out of the box. No flashing necessary.

 

The confusion comes because on a physical hardware level, companies like LSI (Broadcom now) tend to use the exact same hardware on their HBA's vs their RAID cards (to save cost on R&D and retooling for many different yet similar components), and the Firmware is what unlocks the RAID functionality (along with add-ons, such as the Cache and Battery).

 

So, if buying used, you can save a lot of money by buying a cheap RAID Card and flashing the Firmware to it's HBA equivalent. A good example of this is the Dell H200 - it's a "RAID Card", but it shares the same processor and hardware with a bunch of HBA's, so you can cross flash it so it now thinks it's an LSI-9207-8i, for example.

 

In your case, the 9300-8i should work out of the box with zero flashing necessary, since it's packaged and sold as a true HBA, with HBA firmware already loaded on.

 

With that in mind, whether you buy an HBA or a RAID Card depends on how you want to organize your storage. If you opt for FreeNAS (either bare hardware install, or virtualized), HBA is the way to go. Same with ProxMox, since it supports ZFS.

 

If you go for ESXi, you can either use an HBA if you plan on virtualizing FreeNAS (I took this route, it works great). Or you can buy a true RAID Card instead, and just serve out chunks of the RAID array to ESXi to portion out as virtual hard drives. There are pros and cons to both.

 

I like the control and flexibility of ESXi + Virtual FreeNAS + HBA (using PCIe Passthrough to give the entire HBA to the FreeNAS VM). The primary benefit of a hardware RAID card would be simplicity. You create your RAID array in the BIOS (you can install software to manage the RAID card too from the desktop), and that's pretty much that.

[GlorifiedPaperShuffler]

19 minutes ago, dalekphalm said:

Good call to not virtualize it. However, I'd still recommend against doing a full blown pfSense build - as others have pointed out, grabbing a hardware pfSense router is a better idea. You save a lot on the hardware cost, and you can still tinker in full at the software level.

I know the heading said a 2U box for pfsense, but I forgot to remove that. The one Netgate Appliance specified in there is the SG-3100, one step above the basic box someone else had recommended. I saw that it had two M.2 slots for easy upgrading later, if for whatever reason I need to have 64GB of storage on a router. Decided eh, I could upgrade it later, or if I'm doing something that really justifies the extra horsepower, I can always just deal with pfsense virtualization at that point.

23 minutes ago, dalekphalm said:

With that in mind, whether you buy an HBA or a RAID Card depends on how you want to organize your storage. If you opt for FreeNAS (either bare hardware install, or virtualized), HBA is the way to go. Same with ProxMox, since it supports ZFS.

 

If you go for ESXi, you can either use an HBA if you plan on virtualizing FreeNAS (I took this route, it works great). Or you can buy a true RAID Card instead, and just serve out chunks of the RAID array to ESXi to portion out as virtual hard drives. There are pros and cons to both.

 

I like the control and flexibility of ESXi + Virtual FreeNAS + HBA (using PCIe Passthrough to give the entire HBA to the FreeNAS VM). The primary benefit of a hardware RAID card would be simplicity. You create your RAID array in the BIOS (you can install software to manage the RAID card too from the desktop), and that's pretty much that.

I think I might stick to ESXi + Virtual FreeNAS with HBA at this moment. There seems to be a lot more documentation about passing an HBA over to FreeNAS in ESXi, instead of portioning out a RAID array to FreeNAS. It'll make setup easier, so I can actually tinker with the fun stuff, instead of struggling just to get storage up and running at all.

 

P.S. >> Thank you very much for all this free advice you're just giving out like candy. Pretty sure I'd have to pay for a training course or something without you and the LTT community. I'm an automotive development engineer IRL, so feel free to ask me for advice in that realm whenever you need it.

[dalekphalm]

6 hours ago, DaAznKnight said:

I know the heading said a 2U box for pfsense, but I forgot to remove that. The one Netgate Appliance specified in there is the SG-3100, one step above the basic box someone else had recommended. I saw that it had two M.2 slots for easy upgrading later, if for whatever reason I need to have 64GB of storage on a router. Decided eh, I could upgrade it later, or if I'm doing something that really justifies the extra horsepower, I can always just deal with pfsense virtualization at that point.

Ah excellent choice. You can always save the pfSense config and import that into a VM at a later date (or a dedicated build, if you ever find yourself needing the extra horse power).

6 hours ago, DaAznKnight said:

I think I might stick to ESXi + Virtual FreeNAS with HBA at this moment. There seems to be a lot more documentation about passing an HBA over to FreeNAS in ESXi, instead of portioning out a RAID array to FreeNAS. It'll make setup easier, so I can actually tinker with the fun stuff, instead of struggling just to get storage up and running at all.

Indeed, there are plenty of good guides on the topic.

 

I would note, that if you ever decided to go with a Hardware RAID Card, you shouldn't use it with FreeNAS. ZFS loses most of it's primary benefits if used with a RAID Card, since it loses direct hardware level access to the drives.

 

If you did go RAID, you'd be better off swapping FreeNAS out with a Linux or Windows based file server that is just given VHD's portioned from the RAID array, and then shared out via SMB/Samba, etc.

 

With that in mind, I like your current plan.

6 hours ago, DaAznKnight said:

P.S. >> Thank you very much for all this free advice you're just giving out like candy. Pretty sure I'd have to pay for a training course or something without you and the LTT community. I'm an automotive development engineer IRL, so feel free to ask me for advice in that realm whenever you need it.

What exactly does an automotive development engineer do? Sounds intriguing.

 

I'm a SysAdmin and IT Technician for a medium sized Public Library and Art Gallery (5 branches, plus a small satellite branch). We're part of a 3 man (currently 4, since we've got an intern on temporary contract) IT team that literally handles everything from basic desktop/printer support, all the way up to networking and server admin.

 

Either way, you're really welcome. We love server stuff in here and it's great to see people want to learn and jump in.

[GlorifiedPaperShuffler]

2 hours ago, dalekphalm said:

What exactly does an automotive development engineer do? Sounds intriguing.

So I work for one of the mainstream Japanese automakers.

 

Basically, the Japanese name for my field is 設計, or "designer". But "designer" in English doesn't have the same meaning. People generally think fancy sketches of crazy cars when you say "designer".

 

So I'd probably call it "design engineer". Basically, we take what the product planning guys say they want in next-gen ABC vehicle, and do our best to turn it into reality. We'll study the technical requirements, define specifications, determine where to place it in the vehicle, min/max parameters (weight, fuel economy, etc.), issue drawings for manufacturing, do revisions to improve things, file patents, etc.

 

We're what happens between marketing's imaginations and mass production.

 

Specifically, I am in Electronic Platform Design. So that means enabling all the various parts of the car to talk to each other (CAN/LIN communications from physical layer onwards between each part's CAN/LIN interface), enabling the car to talk to other things (think Onstar and internet services, and the associated cybersecurity aspects too), and enabling ancillary stuff like key fobs talking to the car, etc.

[leadeater]

7 hours ago, DaAznKnight said:

So I work for one of the mainstream Japanese automakers.

 

Basically, the Japanese name for my field is 設計, or "designer". But "designer" in English doesn't have the same meaning. People generally think fancy sketches of crazy cars when you say "designer".

 

So I'd probably call it "design engineer". Basically, we take what the product planning guys say they want in next-gen ABC vehicle, and do our best to turn it into reality. We'll study the technical requirements, define specifications, determine where to place it in the vehicle, min/max parameters (weight, fuel economy, etc.), issue drawings for manufacturing, do revisions to improve things, file patents, etc.

 

We're what happens between marketing's imaginations and mass production.

 

Specifically, I am in Electronic Platform Design. So that means enabling all the various parts of the car to talk to each other (CAN/LIN communications from physical layer onwards between each part's CAN/LIN interface), enabling the car to talk to other things (think Onstar and internet services, and the associated cybersecurity aspects too), and enabling ancillary stuff like key fobs talking to the car, etc.

That sounds like a very interesting job 

[jagdtigger]

On 3/15/2019 at 3:38 AM, Bitter said:

There's a company with pfsense hardware using an AMD Jaguar (I think) CPU which passive cools against the case and runs a laptop brick for power.

https://www.pcengines.ch/newshop.php?c=4

I used a APU3C4, its CPU was hitting 80+% utilization(not to mention the RAM) with snort and pfblockerng with a 100 Mb/s internet....  If he wont use any of these then he might get away with it if he doesnt have an overkill internet connection...

[GlorifiedPaperShuffler]

Bumpity bump for a slightly dumb question.

 

I'm slightly revising my purchasing strategy (pulling the trigger in June). I'd like to take a look at used or refurbished components like CPUs and such.

 

I heard about Natex.us, but they appear defunct based on everything just being out of stock, and their Reddit account's last post was almost a year ago.

 

What are the current major stores similar to Natex.us? I mainly want to purchase a board+CPU combo, preferably of the Xeon E5 SR0K variety.

[dalekphalm]

9 hours ago, DaAznKnight said:

Bumpity bump for a slightly dumb question.

 

I'm slightly revising my purchasing strategy (pulling the trigger in June). I'd like to take a look at used or refurbished components like CPUs and such.

 

I heard about Natex.us, but they appear defunct based on everything just being out of stock, and their Reddit account's last post was almost a year ago.

 

What are the current major stores similar to Natex.us? I mainly want to purchase a board+CPU combo, preferably of the Xeon E5 SR0K variety.

Server monkey used to be one of the go-to places for used enterprise equipment. 

 

Ebay would likely be the obvious choice, but you gotta ensure compatibility yourself. 

[Velcade]

Ebay is fine for used server equipment.  I went that route with my new Server setup.

 

If you want something that comes prebuilt you could look at orangecomputers.com

[GlorifiedPaperShuffler]

Follow up question...

 

If I choose a CPU that is affected by the recent Zombieload, Meltdown, and Spectre vulnerabilities, how much of a damn should I give?

[kladzen]

8400T is overkill for a pfsense router... I would save the money and go for the cheapest 8th gen cpu instead.. like 8100T. still 4 cores, 3.1ghz and plenty of speed for a pfsense box...

[mtz_federico]

4 hours ago, DaAznKnight said:

Follow up question...

 

If I choose a CPU that is affected by the recent Zombieload, Meltdown, and Spectre vulnerabilities, how much of a damn should I give?

For pfsense not much for servers just be careful with what you install

[dalekphalm]

6 hours ago, DaAznKnight said:

Follow up question...

 

If I choose a CPU that is affected by the recent Zombieload, Meltdown, and Spectre vulnerabilities, how much of a damn should I give?

Not a lot of options to actually avoid those vulnerabilities. 

 

Just make sure to read up on how to mitigate any risks, proper server security practices, etc. 

[Mikensan]

8 hours ago, DaAznKnight said:

Follow up question...

 

If I choose a CPU that is affected by the recent Zombieload, Meltdown, and Spectre vulnerabilities, how much of a damn should I give?

Mostly an insider threat issue - won't be something somebody does externally. The crazy thing is at how many levels you have to "patch" to be fully protected anyway. Patched firmware still requires your OS be patched which requires your software still be patched. 

 

So imo it's mostly businesses and people with poor browsing habbits that are most at risk. I intentially do not patch my homelab stuff with this shit because I'll miss something, still be vulnerable, and have to suffer with slower speeds.