Homelab start up questions/help on ESXi

[Craigk_c19]

Ultimately I'm looking to build the following servers: 1) Windows 2012 R2 Server (Active Directory) 2) FreeNAS (Storage) 3) pfSense (Firewall/Router) 4) Plex Media Server 5+) Various other VMs to refresh and/or learn Windows Server/AD/Linux/etc training as needed.

 

I had my old gaming PC sitting in the closet and came into some free store and a NIC from work and want to make a home lab. Hardware is a follows

 

CPU: i7 3770k RAM: 24 GBs 1600mhz Motherboard: ASUS Z77 Sabertooth PSU: Evga 750w Gold certified Nic: Intel Pro/1000 Pt Quad Port Store: 6 Seagate 2TB 7200rpm & 4 WD 2TB 7200rpm

I have it in a Fractal Design r6 case so it all fits and I have ESXi 6.5 running off a USB.

 

Now, onto why I'm making this post will what I want to do work? Or is there a better way to set it up? I'm fairly new to this side of everything and looking for any advice/direction. The FreeNAS will be used for shared storage and backups of all our devices. Currently, have roughly 2TBs of movies. Would at some point like to set up a home VPN so we can access our files from anywhere. And since this is all being housed in a Fractal r6 Case my MB only had 8 Stat ports how do I go about attaching all 10 drives correctly? 

 

Any help would be greatly appreciated.

[Dennis Miller]

Are you trying to create all 4+ machines as separate VMs running simultaneously on the same physical system?  If so, then I believe you're really going to be constrained by that CPU.  It only has four cores and single core VMs really crawl.

 

If it's for purely learning purposes, I would suggest only keeping the domain controller running and spin up the others one at a time to play with.

[Craigk_c19]

The 3770k is a 4 core 8 threads, and my plan is currently run ESXi as the hypervisor, and to start have if possible have FreeNas spun up and Plex server. Then plan on learning/trying/playing the other of boxes like PfSense / windows for active directory/test boxes if my set up cant handle that then it will stay as a FreeNas for backups for our PCs. but I'm not sure i can get all my drives up and going to raid them or not? so seeing what options i have today my power supply comes in and ill find out if the on board 8 stat ports i have will even be seen or usable in ESXi, but even then i need to find a way to add the other 2 drives. 

[leadeater]

8 minutes ago, Dennis Miller said:

Are you trying to create all 4+ machines as separate VMs running simultaneously on the same physical system?  If so, then I believe you're really going to be constrained by that CPU.  It only has four cores and single core VMs really crawl.

You can share cores between VMs and the AD and Plex (if not transcoding) won't use much CPU time at all.

[Craigk_c19]

9 minutes ago, Dennis Miller said:

Are you trying to create all 4+ machines as separate VMs running simultaneously on the same physical system?  If so, then I believe you're really going to be constrained by that CPU.  It only has four cores and single core VMs really crawl.

 

If it's for purely learning purposes, I would suggest only keeping the domain controller running and spin up the others one at a time to play with.

The 3770k is a 4 core 8 threads, and my plan is currently run ESXi as the hypervisor, and to start have if possible have FreeNas spun up and Plex server. Then plan on learning/trying/playing the other of boxes like PfSense / windows for active directory/test boxes if my set up cant handle that then it will stay as a FreeNas for backups for our PCs. but I'm not sure I can get all my drives up and going to raid them or not? so seeing what options I have today my power supply comes in and ill find out if the onboard 8 stat ports I have will even be seen or usable in ESXi, but even then I need to find a way to add the other 2 drives. 

[Windows7ge]

I would argue you to put pfSense on it's own box. You will have to take this server offline for maintenance periodically or if something goes wrong and you don't want it taking out the whole home(site).

 

Although FreeNAS can run in a VM (though not recommended) you'll have degraded performance as a result. ZFS (The file system FreeNAS uses) likes direct access to the drives and problems can arise when passing them though to a VM, or worse if you don't pass them through at all. ZFS also utilizes RAM as a read cache of which native access would be better.

 

Everything else I don't have an argument for. I think you could virtualize them.

 

If you plan to use software RAID you'll need an HBA if the motherboard doesn't have enough ports.

[Craigk_c19]

1 minute ago, Windows7ge said:

I would argue you to put pfSense on it's own box. You will have to take this server offline for maintenance periodically or if something goes wrong and you don't want it taking out the whole home(site).

 

Although FreeNAS can run in a VM (though not recommended) you'll have degraded performance as a result. ZFS (The file system FreeNAS uses) likes direct access to the drives and problems can arise when passed them though to the VM, or worse if you don't pass them through at all. ZFS also utilizes RAM as a read cache of which native access would be better.

 

Everything else I don't have an argument for. I think you could virtualize them.

 

If you plan to use software RAID you'll need an HBA if the motherboard doesn't have enough ports.

That's all extremely helpful!! Okay so PfSense off the box entirely got it, and It will not be a good idea to VM FreeNas that is also really good to know. So, for now, I should just use Storage in the pool maybe make a file share on the Windows machine might be my best bet currently until I get more hardware to implement it correctly.   

 

As for the HBA do you have any suggestions? I've tried looking into them last night and got lost very quickly 

[Windows7ge]

3 minutes ago, Craigk_c19 said:

As for the HBA do you have any suggestions? I've tried looking into them last night and got lost very quickly 

I've been using three LSI 9207-8i's. They use the SFF-8087 connector. It gives you eight SATAIII ports.

 

I have to say do NOT buy it from what I just linked. That's where I bought it from a few years back but they cost me $82/each. That listed price is a rip-off so definitely search elsewhere if that HBA interests you.

 

If you need 16 ports I know of another card that has been working well for me thus far.

[Craigk_c19]

8 minutes ago, Windows7ge said:

I've been using three LSI 9207-8i's. They use the SFF-8087 connector. It gives you eight SATAIII ports.

 

I have to say do NOT buy it from what I just linked. That's where I bought it from a few years back but they cost me $82/each. That listed price is a rip-off so definitely search elsewhere if that HBA interests you.

 

If you need 16 ports I know of another card that has been working well for me thus far.

would you mind linking the 16 port one as well? 8 ports would only work if i use 8 on the HBA and then 2 through the Motherboard. even on Amazon that LSI 9207-8is still run roughly  $95ish

[Windows7ge]

3 minutes ago, Craigk_c19 said:

would you mind linking the 16 port one as well? 8 ports would only work if i use 8 on the HBA and then 2 through the Motherboard. even on Amazon that LSI 9207-8is still run roughly  $95ish

I bought my three 16 port cards off eBay of which eBay links go against the forums Community Standards so I can't give you a URL. I can tell you though that they are the LSI 9201-16i. For my setup they have been plug'n'play. No issues. I think I paid like ~$130 each and hey $95 is significantly better than the Newegg price right now.

 

I don't know what file system ESXi uses but maybe @leadeater can comment here. I can say ZFS doesn't care what controller the drives are plugged into. It'll RAID them together. All it cares about is direct access so you could have some running off the motherboard South Bridge/PCH, some running off a unconfigured marvel controller, some off an dedicated HBA, ZFS doesn't care. I'd like to imagine it's a similar situation with ESXi...*looks at leadeater*...

[Craigk_c19]

8 minutes ago, Windows7ge said:

I bought my three 16 port cards off eBay of which eBay links go against the forums Community Standards so I can't give you a URL. I can tell you though that they are the LSI 9201-16i. For my setup they have been plug'n'play. No issues. I think I paid like ~$130 each and hey $95 is significantly better than the Newegg price right now.

 

I don't know what file system ESXi uses but maybe @leadeater can comment here. I can say ZFS doesn't care what controller the drives are plugged into. It'll RAID them together. All it cares about is direct access so you could have some running off the motherboard South Bridge/PCH, some running off a unconfigured marvel controller, some off an dedicated HBA, ZFS doesn't care. I'd like to imagine it's a similar situation with ESXi...*looks at leadeater*...

I'll wait until leadeater chimes in but Ebay looks to be the way to go the LSI 9201-16i is $120ish and LSI 9207-8i is $40-60 ish can't see any reason not to just get 2 9207-8is i have the PCIE slots for them. But if it is like you said I would only need one then use the onboard 2 Intel and 2 AsMedia Sata 3 ports.

Would it work if I used the Bare bone to put FreeNas on then put ESXi on top of that? Or just stick strictly to have a dedicated FreeNas box? 

sorry, I'm so new to this.

[Windows7ge]

7 minutes ago, Craigk_c19 said:

Would it work if I used the Bare bone to put FreeNas on then put ESXi on top of that? Or just stick strictly to have a dedicated FreeNas box? 

FreeNAS's VM software (Bhyve) is garbage and to install a hypervisor inside a VM (a bad one at that) would spell catastrophe for everything. Hypervisor on the hardware, everything else on the hypervisor.

 

Assuming the FreeNAS box isn't going to do anything seriously workload intensive besides a file share, maybe SSH/SFTP and some light media for plex you won't need a strong/expensive system. You could build it compact with a Micro-ATX or Mini-ITX motherboard, mainstream CPU socket (Xeon), and maybe 8GB of ECC RAM. It'd work a treat.

[Craigk_c19]

33 minutes ago, Windows7ge said:

FreeNAS's VM software (Bhyve) is garbage and to install a hypervisor inside a VM (a bad one at that) would spell catastrophe for everything. Hypervisor on the hardware, everything else on the hypervisor.

 

Assuming the FreeNAS box isn't going to do anything seriously workload intensive besides a file share, maybe SSH/SFTP and some light media for plex you won't need a strong/expensive system. You could build it compact with a Micro-ATX or Mini-ITX motherboard, mainstream CPU socket (Xeon), and maybe 8GB of ECC RAM. It'd work a treat.

You hit the nail on the head, that's all I'm looking to do with it and have it be a daily/weekly backup on all machines in the house in case something happens. Does it need to be a Xeon and have ECC Ram though? that seems overkill I feel like the box won't be getting hard I wouldn't think other than maybe to stream a movie to the living room tv and maybe 1 other room at the same time. Would a newer chip work like the ryzen 1700 8 cores 16 threads can get those for under 100 now.
 

[Windows7ge]

19 minutes ago, Craigk_c19 said:

You hit the nail on the head, that's all I'm looking to do with it and have it be a daily/weekly backup on all machines in the house in case something happens. Does it need to be a Xeon and have ECC Ram though? that seems overkill I feel like the box won't be getting hard I wouldn't think other than maybe to stream a movie to the living room tv and maybe 1 other room at the same time. Would a newer chip work like the ryzen 1700 8 cores 16 threads can get those for under 100 now.
 

The Xeons are recommended because it's a standard (you can get away with using AMD if you want) but the ECC RAM is a real recommendation. ZFS works with ECC memory to ensure the correct data get written to the pool without error. ECC memory is only marginally more expensive than non-ECC anyways so you might as well take advantage of it. AMD CPUs do support ECC memory but it's been a while since I've used an AMD platform so I'm unsure if all AMD socket motherboards support it too.

[Mikensan]

Just wanted to chime in and say the 4 cores will work just fine for a lab. Given most of your VMs will be idle (except plex) you'll actually get use of the HT'd cores, once Plex pegs a couple to transcode a movie then those HT cores won't mean much. So I would only assign 2-3 cores to Plex leaving 1 available for the rest of your VMs which should be able to work with the 1 + HT to do the light chores they have to.

 

By contrast I probably have 10 VMs running off of 12 cores (2x 5670s) each assigned 2 cores. My environment runs really well, no complaints.

[leadeater]

7 hours ago, Craigk_c19 said:

would you mind linking the 16 port one as well? 8 ports would only work if i use 8 on the HBA and then 2 through the Motherboard. even on Amazon that LSI 9207-8is still run roughly  $95ish

That will work fine though you'll have to pass through the entire SATA controller and SAS HBA to the FreeNAS VM, that means using a USB drive for the ESXi install which I recommend anyway.

 

Once FreeNAS is setup you'll need to create an NFS share then mount that on the ESXi host so you can create VMs, because until now you have no storage you can use for ESXi at all. This also means you need to use one of the HDDs for FreeNAS OS itself or, and personally I'd do it this way, not pass through the motherboard SATA controller and use a small SSD or HDD for an ESXi datastore. You'll need a SAS HBA with enough ports for all the HDDs that FreeNAS will use that will get passed through to the VM.

 

Basically you need some kind of storage for ESXi to use to store the VM configuration files as well as any potential virtual disks, chicken and egg problem. You never want to put virtual disks on a USB drive, way too slow, which is why I'm recommending not to pass through the motherboard SATA controller.

 

Another important note is if you do create an NFS share on the FreeNAS server and mount it on the ESXi server any VMs stored on that cannot boot until the FreeNAS server is booted and the ESXi server has remounted the NFS share (will happen automatically).

[Craigk_c19]

13 hours ago, leadeater said:

That will work fine though you'll have to pass through the entire SATA controller and SAS HBA to the FreeNAS VM, that means using a USB drive for the ESXi install which I recommend anyway.

 

Once FreeNAS is setup you'll need to create an NFS share then mount that on the ESXi host so you can create VMs, because until now you have no storage you can use for ESXi at all. This also means you need to use one of the HDDs for FreeNAS OS itself or, and personally I'd do it this way, not pass through the motherboard SATA controller and use a small SSD or HDD for an ESXi datastore. You'll need a SAS HBA with enough ports for all the HDDs that FreeNAS will use that will get passed through to the VM.

 

Basically you need some kind of storage for ESXi to use to store the VM configuration files as well as any potential virtual disks, chicken and egg problem. You never want to put virtual disks on a USB drive, way too slow, which is why I'm recommending not to pass through the motherboard SATA controller.

 

Another important note is if you do create an NFS share on the FreeNAS server and mount it on the ESXi server any VMs stored on that cannot boot until the FreeNAS server is booted and the ESXi server has remounted the NFS share (will happen automatically).

Thank you, that's some seriously detailed information! After some more digging yesterday I found that running FreeNas on a VM might not work that great. So currently looking into building that out then just having an ESXi box with the listed hardware minus the drives, does that seem like a better idea to you? Then as you suggested, I can slap a 250GB SSD into the ESXi box and that should be enough correct? I

 

If I go this route I won't have to worry about passing through any SATA controllers and the HBA can just be in the FreeNAS box. The only reason I'm able to do a FreeNAS build now is I came across a free i7-6700 recently and I'm selling it on eBay for 240 and with that, I'm able to get an Ivy bridge Xeon with the motherboard and 16 GBs of ECC ram along with 1 HBA  LSI 9207-8i. Will just have to get another PSU out of pocket which I can swing. 

 

Seems this is the direction a lot of people have suggested to me in the past just didn't think it was possible until now.

[leadeater]

6 hours ago, Craigk_c19 said:

After some more digging yesterday I found that running FreeNas on a VM might not work that great. So currently looking into building that out then just having an ESXi box with the listed hardware minus the drives, does that seem like a better idea to you? Then as you suggested, I can slap a 250GB SSD into the ESXi box and that should be enough correct? I

 

If I go this route I won't have to worry about passing through any SATA controllers and the HBA can just be in the FreeNAS box. The only reason I'm able to do a FreeNAS build now is I came across a free i7-6700 recently and I'm selling it on eBay for 240 and with that, I'm able to get an Ivy bridge Xeon with the motherboard and 16 GBs of ECC ram along with 1 HBA  LSI 9207-8i. Will just have to get another PSU out of pocket which I can swing. 

Yea sure if you can do more than one server then that is more ideal. VM hardware passthrough works but it's a bit of a pain as you can't edit the VM configuration with it powered on if you need to and administration wise having your host dependent on a VM itself is running is not great. As an example I did something stupid a while ago by mistake, I have a similar setup but a Windows storage server and I moved the VM to the datastore it was presenting to the host and some time later shut the VM down..... yep that means it was impossible to power back on lol. Don't do that, don't be stupid like I was. I had a reason why I did it, but I forgot that I moved it and it needed moving back to the local SSD.

[Craigk_c19]

One thing, as a human, we learn pretty well from our mistakes. I bet you are fully aware of that now and anytime you do have to shut it down you double check, to make sure it doesn't happen again. I would like to say I hope I never make that mistake, but if I do I bet i only do it once haha 

[leadeater]

9 hours ago, Craigk_c19 said:

One thing, as a human, we learn pretty well from our mistakes. I bet you are fully aware of that now and anytime you do have to shut it down you double check, to make sure it doesn't happen again. I would like to say I hope I never make that mistake, but if I do I bet i only do it once haha 

What made it worse was I knew about it, and knew that would happen, but forgot I moved the VM to that storage. Be ware of temporary measures, they often become long term and you forget 

[Rusted]

I have been building my own homelab. Currently I have HPE proliant running freeNAS and nextcloud is running on top of that. Im going to use old lenovo desktop pc at first for VMs but that is still in works. I will keep my NAS as separate box as I use it daily.

Then I have pcengines APU running my pfsense router. I wouldn't run a pfsense on VM if it's used for networking, messing up routers or VMs config might cause your network go down and make repairing it troublesome. Pfsense router handles my internet connection and takes the beating from port scanning, etc. So it as a separate box is also a security thing.

[dalekphalm]

On 4/16/2019 at 11:58 AM, Craigk_c19 said:

-snip-

I'd just like to chime in re: FreeNAS VM.

 

The main reason people "discourage" FreeNAS VM's is because ZFS (the file system and it's associated RAID-like capabilities) likes to have direct access to your storage devices (HDD/SSD, etc).

 

However, assuming your hardware supports it, this is actually easy enough to do. To solve the problem, you simply need to pass your storage controller (usually an HBA, but could be something as simple as a few SATA expansion cards) directly to the VM. This is called PCI Passthrough (though I refer to it as PCIe Passthrough most of the time, due to.. well... no one uses PCI anymore).

 

I use ESXi in my Home Server, and have an LSI 9207-8e HBA Card (Normally people would use the 8i version, but I have a Dell MD1200 DAS (Direct Attach Storage) Disk Shelf). This HBA is passed directly to the FreeNAS VM via PCI Passthrough (6x Toshiba 3TB SATA HDD's).

 

FreeNAS then sees the drives directly, just as if I had installed FreeNAS directly onto the system.

 

It works fantastic. I also have experienced no RAM related issues.

 

Another note on FreeNAS: You see EVERYWHERE on the web that people recommend 1GB of RAM per 1TB of storage. Ignore this. It's entirely wrong advice. That "rule of thumb" was specifically for use of Deduplication (a technique that scans the storage array for identical files, keeps only one of them, and then uses pointers for the other copies). Dedupe doesn't work great on FreeNAS anyway, and I would never advocate anyone using it here.

 

So in terms of RAM, 4GB is good, 8GB is better. It'll run on less than 4GB of RAM though, but performance might suffer below 4GB.

 

Anything above 8GB on FreeNAS usually isn't warranted, unless you're doing advanced features or non-standard things.

[Craigk_c19]

20 hours ago, dalekphalm said:

I'd just like to chime in re: FreeNAS VM.

 

The main reason people "discourage" FreeNAS VM's is because ZFS (the file system and it's associated RAID-like capabilities) likes to have direct access to your storage devices (HDD/SSD, etc).

 

However, assuming your hardware supports it, this is actually easy enough to do. To solve the problem, you simply need to pass your storage controller (usually an HBA, but could be something as simple as a few SATA expansion cards) directly to the VM. This is called PCI Passthrough (though I refer to it as PCIe Passthrough most of the time, due to.. well... no one uses PCI anymore).

 

I use ESXi in my Home Server, and have an LSI 9207-8e HBA Card (Normally people would use the 8i version, but I have a Dell MD1200 DAS (Direct Attach Storage) Disk Shelf). This HBA is passed directly to the FreeNAS VM via PCI Passthrough (6x Toshiba 3TB SATA HDD's).

 

FreeNAS then sees the drives directly, just as if I had installed FreeNAS directly onto the system.

 

It works fantastic. I also have experienced no RAM related issues.

 

Another note on FreeNAS: You see EVERYWHERE on the web that people recommend 1GB of RAM per 1TB of storage. Ignore this. It's entirely wrong advice. That "rule of thumb" was specifically for use of Deduplication (a technique that scans the storage array for identical files, keeps only one of them, and then uses pointers for the other copies). Dedupe doesn't work great on FreeNAS anyway, and I would never advocate anyone using it here.

 

So in terms of RAM, 4GB is good, 8GB is better. It'll run on less than 4GB of RAM though, but performance might suffer below 4GB.

 

Anything above 8GB on FreeNAS usually isn't warranted, unless you're doing advanced features or non-standard things.

Dang If I would have found/ saw this a few days ago would have been lovely haha because I'm currently building a FreeNAS box now too. My spear i7-6700 is on sale right now on eBay, and I planned on buying an old Xeon and old super micro MB and 16 GB or ECC ddr3 ram all for around 200ish and LSI 9207-8i for 55 and just have a FreeNAS box and an ESXi box and maybe mess with ISCSI to get them to talk. Now debating do I make a proper FreeNAS box or just get the 9207-8i and passthrough and pocket the extra cash... hmmm

[dalekphalm]

On 4/19/2019 at 4:57 PM, Craigk_c19 said:

Dang If I would have found/ saw this a few days ago would have been lovely haha because I'm currently building a FreeNAS box now too. My spear i7-6700 is on sale right now on eBay, and I planned on buying an old Xeon and old super micro MB and 16 GB or ECC ddr3 ram all for around 200ish and LSI 9207-8i for 55 and just have a FreeNAS box and an ESXi box and maybe mess with ISCSI to get them to talk. Now debating do I make a proper FreeNAS box or just get the 9207-8i and passthrough and pocket the extra cash... hmmm

I'd just get the 9207 (fixed typo) and pocket the rest, if it were me.

 

Keep in mind you don't specifically need to use that card. You can often save some money with an OEM branded one like the IBM m1015 or the Dell H200 - though with these, you'll need to flash them.

 

I opted to get the 9207 specifically so that I wouldn't have to deal with flashing the card.

[Mikensan]

On 4/19/2019 at 6:55 PM, dalekphalm said:

I'd just get the 9207 (fixed typo) and pocket the rest, if it were me.

 

Keep in mind you don't specifically need to use that card. You can often save some money with an OEM branded one like the IBM m1015 or the Dell H200 - though with these, you'll need to flash them.

 

I opted to get the 9207 specifically so that I wouldn't have to deal with flashing the card.

Just wanted to add that a lot of ebay sellers now pre-flash their LSI varients without any added cost. I actually just bought a m1015 cheaper than what I paid years ago and it was already updated and flashed to IT mode.