Home Lab AIO Server??

[Guest]

First and foremost, hello everyone, it's my first post here!

 

I am in the midst of planning out a home server lab and wanted to get some clarification and opinions on some options/scenarios that I am thinking about.

 

This a project that I plan on starting later this year and I just want to make sure that I am on the right track as I plan this out. I have done quite a bit of reading and for the most part I have a pretty good idea of what I want to achieve. I have recently switched over to Linux (PopOS) from Windows and have really enjoyed the experience. It's actually one of the main reasons as to why I want to have my own home lab. I already decided that I am going to build a Ryzen Pfsense firewall to replace my traditional consumer grade Linksys router. I have also been wanting to build and NAS and after much thought and consideration I decided that I will build a Unraid server. Along with it being a NAS, I plan using Plex, Nextcloud and OpenVPN as well.

 

As I mentioned earlier, I have recently made the switch over to Linux, but there are still some things I still require Windows for and this is where I would like to get some clarification and opinions. Originally I planned to build myself a new gaming PC, which will also act as my HTPC (this is my current setup). I was going load it up with PopOS and dual boot with Windows, but after some reading I came across VFIO which got me thinking. After some more reading and going down the rabbit hole of endless opportunities/configurations, I learned that Unraid can do all of this in one box, which is really appealing to me. 

 

My first question is whether I should stick with my original plan of having a separate NAS (which also does Nextcloud, Plex and OpenVPN) and a separate gaming/HTPC (either dual booting, or VFIO passthrough), or using Unraid as an AIO solution and just having a separate HTPC? This now leads me into my second question....Originally I was planning on going with the 3700X for dual booting, but have decided that if I use VFIO I would get the 3950X as I want to give the gaming VM at least 6-8 cores. I was also going to use the same CPU for my AIO Unraid server, but I was wondering if it was worth jumping up to the 3960X Threadripper, or will the 3950X be enough to run Unraid with a few dockers, a Linux VM (passthrough) as well as a Windows VM (passthrough)? 

 

I know that NVME drives can be passed through to the VMs, so I plan on having two 1TB NVME drives, one for Linux and one for Windows. This will also apply if I go the AIO route with Unraid. I have read and watched some videos on this topic, and the ones I have seen thus far only talk about giving the NVME back to the Unraid sever once the VM is off, however I do not want this, I want the VMs to have those drives permanently. I was wondering if that is possible, or do they always go back to the Unraid server as unassigned (I will have separate SSDs as cache drives)? There was also talk about potentially overwriting data on those drives, but I figure if they are permanently tied to the VMs, that shouldn't be an issue. I only plan to have non critical data (games and such), so really if anything happens it's not the end of the world. Documents, videos, music, etc.. will all be stored on the Unraid array separately away from VMs.

 

Finally I read about VM servers such as Proxmox, XPC-NG, etc... and was wondering if they really add any benefit over Unraid? From what I understand, Unraid handles all of this quite well, so IMHO it seems redundant to have a Proxmox VM server, but then again I am very new to all of this

 

I understand that some of this really will come down to personal preference, but I am very much a planner, so I like to know what my plan is before I dive in. I just want to make sure that I am not over buying anything (ie getting the Threadripper when the 3950X wil be sufficient), nor do I want to overthink all of this. As I said above, I like the AIO idea, but if it's easier/better to separate things a little, I am all for that as well.

 

Thanks!

[Electronics Wizardy]

28 minutes ago, Spec7re said:

My first question is whether I should stick with my original plan of having a separate NAS

I like the seprate nas/server as its normally simpler to setup. Thats what I personally do.

 

a

29 minutes ago, Spec7re said:

Finally I read about VM servers such as Proxmox, XPC-NG, etc... and was wondering if they really add any benefit over Unraid? From what I understand, Unraid handles all of this quite well, so IMHO it seems redundant to have a Proxmox VM server, but then again I am very new to all of this

Other hypervisors normally give you more control over vms, and more features. I normally stay away from unraid as a hypervisor. Normally other hypervisors give you move detailed control, and more features like clustering if needed. Also Im not a fan of the performance of the unraid array, and like the zfs features.

 

33 minutes ago, Spec7re said:

My first question is whether I should stick with my original plan of having a separate NAS (which also does Nextcloud, Plex and OpenVPN) and a separate gaming/HTPC (either dual booting, or VFIO passthrough), or using Unraid as an AIO solution and just having a separate HTPC? This now leads me into my second question....Originally I was planning on going with the 3700X for dual booting, but have decided that if I use VFIO I would get the 3950X as I want to give the gaming VM at least 6-8 cores. I was also going to use the same CPU for my AIO Unraid server, but I was wondering if it was worth jumping up to the 3960X Threadripper, or will the 3950X be enough to run Unraid with a few dockers, a Linux VM (passthrough) as well as a Windows VM (passthrough)? 

 

What all is the nas running? For those listed uses, unless you need lots of plex transcodes, that won't need much cpu power at all. Normally your ram and disk limited in vms well before cpu limits. So a 3900x or a 3700x should be fine here.

30 minutes ago, Spec7re said:

I know that NVME drives can be passed through to the VMs,

They can be, but I like to use virtual disks for vms as it allows snapshots, so you can easily revert issues.

 

 

[Windows7ge]

What I would do is have your UnRAID server separate from your desktop. You can game over a network by using services such as PARSEC but personally running a Windows VM on a PROXMOX server with GPU pass-though I didn't have a great experience on my Ubuntu 19.04 setup.

 

I have a full tutorial on VFIO in how you could setup your PopOS install to host a Windows VM with GPU pass-though. I based mine on Ubuntu 19.04 but it can easily be adapted for PopOS:

I believe this would be cheaper/simpler since it just requires a second GPU instead of building a whole separate box to then run Windows in a VM over the network. The AM4 series CPU's should be fine. Most games still only really utilize 4~6 cores so even a 8 core/16 thread Ryzen with 16 or 32GB of RAM would run this machine just fine. You don't need to go TR4/40 unless you just want it for enthusiast reasons.

 

I can't speak on behalf of how UnRAID's hypervisor works but for QEMU/KVM if the VM isn't running it doesn't matter if the storage device goes back to the host OS. If you block the drives driver in PopOS then the host won't even touch it during operation it'll just hold it there waiting for your VM to start.

 

PROXMOX vs UnRAID it's really a matter of preference and if you do or don't want to spend money. Personally PROXMOX is an amazing hypervisor. One I've written a beginners guide on. It's free, does have a paid subscription that is optional but it's otherwise free to use. Very user intuitive. Tons of feature. PROXMOX and QEMU/KVM+Virt-manager are my go-to's for virtualization and hardware pass-through.

 

If I missed any of your questions let me know.

[Guest]

10 hours ago, Electronics Wizardy said:

I like the seprate nas/server as its normally simpler to setup. Thats what I personally do.

 

Thanks!

 

I agree, the more I think about it, the more I feel that keeping certain aspects separate might be the better way to go. I mean AIO is nice, but at the same time poses additional challenges and increases the chances of something going wrong. I will more than likely build a NAS and then keep the Gaming/HTPC separate from it. 

 

10 hours ago, Electronics Wizardy said:

Other hypervisors normally give you more control over vms, and more features. I normally stay away from unraid as a hypervisor. Normally other hypervisors give you move detailed control, and more features like clustering if needed. Also Im not a fan of the performance of the unraid array, and like the zfs features.

I haven't looked at FreeNas in a while, but I may have to take another look. Ultimately what drew me to Unraid was the ability to mix and match HDD sizes, but I can probably use the money saved by not going with Threadripper and buy like 4-5 8TB HDDs, load up FreeNas and be done with it. Down the road I can always build a VM server based off Proxmox and let that handle my VMs.

 

10 hours ago, Electronics Wizardy said:

What all is the nas running? For those listed uses, unless you need lots of plex transcodes, that won't need much cpu power at all. Normally your ram and disk limited in vms well before cpu limits. So a 3900x or a 3700x should be fine here.

 

I am glad that a 3700X and a 3900X are more than enough for this application. I may just end up using those CPUs in my gaming/HTPC and use something like a Ryzen 5 3400G for the server. I will try plex like that and see how it does. It will more than likely only be me playing videos, so I probably doesn't need a tone of trans-coding capabilities, but if I run into any issues, I can always pop in a more beefy GPU to handle that load.

 

 I will more than likely wait until the 4th Gen Ryzen CPUs are released to see what they bring to the table. I'll either get a 4th gen, or just snag up a 3900X, or 3950X when the prices come down as the 4th gen is being released. I am more interested in seeing if the new X670 chipsets will run cooler, or have passive cooling vs active, but we will have to wait and see.

 

10 hours ago, Electronics Wizardy said:

They can be, but I like to use virtual disks for vms as it allows snapshots, so you can easily revert issues.

 

I never thought of snapshots, but it's a good point to consider. My line of thinking (and I could have been thinking wrong) is that if I pass through the NVME I will get full performance out of it. That being said, if I use the NVMEs like virtual disks, is the performance pretty much the same since they are so fast? By the time I build this, I will more than likely be using PCIE Gen 4 NVMEs, which by that point are so fast that it may not matter if they are passed through or not. Would I notice any performance impact by using virtual disks, vs passing them through when using gen 4 NVME drives?

9 hours ago, Windows7ge said:

What I would do is have your UnRAID server separate from your desktop. You can game over a network by using services such as PARSEC but personally running a Windows VM on a PROXMOX server with GPU pass-though I didn't have a great experience on my Ubuntu 19.04 setup.

Thanks!

 

As I said above, I think both of your are correct. Keeping things separate may be a much better option than trying to cram everything into an AIO solution. As nice as it would be, it will be more of a pain the more I think about it. I may just try to do VFIO on PopOS first, if that doesn't work out as well, I will try to get a separate Proxmox server up and running and do it that way.

 

10 hours ago, Windows7ge said:

I have a full tutorial on VFIO in how you could setup your PopOS install to host a Windows VM with GPU pass-though. I based mine on Ubuntu 19.04 but it can easily be adapted for PopOS

 

I believe this would be cheaper/simpler since it just requires a second GPU instead of building a whole separate box to then run Windows in a VM over the network. The AM4 series CPU's should be fine. Most games still only really utilize 4~6 cores so even a 8 core/16 thread Ryzen with 16 or 32GB of RAM would run this machine just fine. You don't need to go TR4/40 unless you just want it for enthusiast reasons.

 

Thanks for the video I will definitely take a look! I too believe this may be the better approach for me. I really didn't want to go Threadripper, but as I was reading the more I was thinking about it, but in all honestly I am happy that the Ryzen CPUs (3700X and up) are more than enough.

10 hours ago, Windows7ge said:

I can't speak on behalf of how UnRAID's hypervisor works but for QEMU/KVM if the VM isn't running it doesn't matter if the storage device goes back to the host OS. If you block the drives driver in PopOS then the host won't even touch it during operation it'll just hold it there waiting for your VM to start.

 

PROXMOX vs UnRAID it's really a matter of preference and if you do or don't want to spend money. Personally PROXMOX is an amazing hypervisor. One I've written a beginners guide on. It's free, does have a paid subscription that is optional but it's otherwise free to use. Very user intuitive. Tons of feature. PROXMOX and QEMU/KVM+Virt-manager are my go-to's for virtualization and hardware pass-through.

 

If I missed any of your questions let me know.

Thanks for the clarification!

 

I was just getting confused because people were only talking about certain scenarios, but didn't take the time to explain it fully. 

 

Proxmox is growing on me, the more I hear/read about it. I really would like to eventually have a Proxmox setup, so I guess if the VFIO passthrough on PopOS doesn't work out, I may just take that computer, make into a Proxmox server and build a lower powered HTPC and just do it that way. 

 

Still lots to think about!

[Windows7ge]

2 hours ago, Spec7re said:

Thanks!

 

As I said above, I think both of your are correct. Keeping things separate may be a much better option than trying to cram everything into an AIO solution. As nice as it would be, it will be more of a pain the more I think about it. I may just try to do VFIO on PopOS first, if that doesn't work out as well, I will try to get a separate Proxmox server up and running and do it that way.

Assuming you're on a modern version of PopOS and your desktop supports enabling IOMMU this should work without issue. Looking Glass is a great tool for making it so you don't need a separate keyboard, mouse, monitor to use the Guest OS.

 

2 hours ago, Spec7re said:

Thanks for the clarification!

 

I was just getting confused because people were only talking about certain scenarios, but didn't take the time to explain it fully. 

 

Proxmox is growing on me, the more I hear/read about it. I really would like to eventually have a Proxmox setup, so I guess if the VFIO passthrough on PopOS doesn't work out, I may just take that computer, make into a Proxmox server and build a lower powered HTPC and just do it that way. 

 

Still lots to think about!

My beginner's guide is already outdated due to updates to PROXMOX but much of it should still be relevant. You may find section 9 most helpful.

 

[Electronics Wizardy]

3 hours ago, Spec7re said:

never thought of snapshots, but it's a good point to consider. My line of thinking (and I could have been thinking wrong) is that if I pass through the NVME I will get full performance out of it. That being said, if I use the NVMEs like virtual disks, is the performance pretty much the same since they are so fast? By the time I build this, I will more than likely be using PCIE Gen 4 NVMEs, which by that point are so fast that it may not matter if they are passed through or not. Would I notice any performance impact by using virtual disks, vs passing them through when using gen 4 NVME drives?

Ye you will lose some performance, but normally the extra io isn't needed. Some nvme drives can be split to be passthrough to multiple vms, but for home uses, the flexability is probalby better than the extra speed.

 

 

[Guest]

On 4/11/2020 at 11:01 AM, Windows7ge said:

Assuming you're on a modern version of PopOS and your desktop supports enabling IOMMU this should work without issue. Looking Glass is a great tool for making it so you don't need a separate keyboard, mouse, monitor to use the Guest OS.

 

My beginner's guide is already outdated due to updates to PROXMOX but much of it should still be relevant. You may find section 9 most helpful.

 

Thanks for the guides.

 

I have had a quick read through and they are very detailed and will help me greatly as start this process!

23 hours ago, Electronics Wizardy said:

Ye you will lose some performance, but normally the extra io isn't needed. Some nvme drives can be split to be passthrough to multiple vms, but for home uses, the flexability is probalby better than the extra speed.

 

 

I am ok with that, as long as I can still get some very good performance I will be happy. As I said earlier, NVME drives are so fast that in real world usage I probably won't notice a difference anyways, especially with Gen 4 pci-e NVEM drives.