Virtualisation home server build

[Amadea]

Hi everyone, long time subscriber, first time posting. I am looking for some advise on buying hardware for a new virtualisation server at home. The problem is that the last time I was intensively involved in PC building, was in the Pentium age..

 

Currently I have multiple Raspberry Pi's in a server case, each running their own distro and services. This seemed like a good idea since these ARM chips are rather energy efficient and contain just the right amount of processing power needed for each service. A couple years later the pile grew and it seems impracticable at this stage. So I was thinking of buying a single up-to-date x86-64 system for the years ahead and running Proxmox. To give an idea of what I'm planning to run:

1. [VM] Some NAS OS
2. [VM] Blue Iris
3. [VM] TVheadend
4. [VM] FreePBX
5. [VM or Container] MySQL server
6. [VM] OpenHAB
7. [Container] PiHole
8. [Container] MQTT broker
9. [VM or Container] Syslog server
10. [Container] Audio server/bridge (multiple, but running incidentally)
11. [VM] Plex (maybe, and only if it plays nicely with TVheadend)

I'm planning to have a small M.2 SSD for Proxmox to be installed on. And then multiple SATA HDD's in a (software) RAID where 1 drive may fail. I would like the whole RAID space to be available through a NAS OS VM on top of Proxmox, for storage of the VM disks, ISO's and media files for TVheadend (and maybe Plex). I'm actually not sure if this approach is possible or efficient, so any advise on this matter would be appreciated. My goal is to have a single (software) RAID for everything except the Proxmox OS itself. So: ISO images, VM disks, Containers, NVR recording (Blue Iris), DVR recording (TVheadend), other streaming media files, backups of local PC's, general network shared files.

 

Then their is the matter of processor specific functionalities. For Blue Iris and TVheadend I would like to use a CPU-integrated video encoder, like Quick Sync for Intel. I've read that passing such functionalities onto a single VM is possible. But I'm not sure if both VM's at the same time is possible. AES-NI would also be nice to be available to all VM's and containers. Furthermore I have, at this point in time, no plans to pass any PCIe to VM's, but passing a specific USB device is required.

 

So for the hardware I need some advise on the CPU and ATX motherboard, either Intel or AMD. Server case, fans, PSU, RAM (non-ECC), SATA expansion card are taken care of. I can however use some advise on the 2.5" SATA RAID drives for longevity (especially since they are also used for a NVR). My general goal is to make it energy efficient as possible, but also with some threads to spare for incidental VM spinups. I was thinking about 6/8 cores, 12/16 threads, since you can share threads between VM's in Proxmox and most systems are near idle most of the time. For just the CPU and motherboard I'm planning to not spend more then 500.00 USD or 400.00 EUR. What do you think about my plans and what hardware would you recommend?

 

[WereCatf]

2 minutes ago, Amadea said:

I'm actually not sure if this approach is possible or efficient

It is entirely possible.

3 minutes ago, Amadea said:

My goal is to have a single (software) RAID for everything except the Proxmox OS itself. So: ISO images, VM disks, Containers, NVR recording (Blue Iris), DVR recording (TVheadend), other streaming media files, backups of local PC's, general network shared files.

You can't both store the VMs on the RAID and run them off the RAID, if the NAS OS is the one that's supposed to be handling the RAID.

5 minutes ago, Amadea said:

For Blue Iris and TVheadend I would like to use a CPU-integrated video encoder, like Quick Sync for Intel. I've read that passing such functionalities onto a single VM is possible. But I'm not sure if both VM's at the same time is possible

Intel's iGPU can be shared between multiple VMs, yes. I am using one such setup myself. That said, I don't know how to set such a system up in Proxmox. Google around for GVT-G or look at e.g. https://wiki.archlinux.org/index.php/Intel_GVT-g

6 minutes ago, Amadea said:

AES-NI would also be nice to be available to all VM's and containers

That's a CPU-feature. As long as you're passing through the correct type of CPU to the VMs, they'll be able to use it -- typically you'd just set the CPU-type the same as the host-model.

7 minutes ago, Amadea said:

either Intel or AMD

AMD iGPU cannot be shared between VMs, so that rules that out.

[Amadea]

@WereCatfThank you allot for your answers! Especially about the video encoding.

7 minutes ago, WereCatf said:

You can't both store the VMs on the RAID and run them off the RAID, if the NAS OS is the one that's supposed to be handling the RAID.

Let me put it differently; I want all* VM's to be able to access "a shared piece of space" were all* of them can read and write media files. And this space should also be accessible by PC's inside the LAN through Samba for example, to play a media file or upload a new ISO. Any advise?

[WereCatf]

1 minute ago, Amadea said:

Let me put it differently; I want all* VM's to be able to access "a shared piece of space" were all* of them can read and write media files. And this space should also be accessible by PC's inside the LAN through Samba for example, to play a media file or upload a new ISO. Any advise?

Proxmox itself already has support for ZFS, which means you can setup a RAID from Proxmox directly, if you wish. Similarly, Proxmox can also expose the RAID's contents via Samba directly.

 

If you do wish to use a proper NAS OS, on the other hand, you'd have to store the VMs separately from the shared space that all the VMs can access. How you do that is up to you, but e.g. buying a bigger SSD for Proxmox and the VMs would be one option. Kind of depends on how many drives you have, how much fault-tolerance you want and how many drives you can connect to the system to begin with.

[Amadea]

44 minutes ago, WereCatf said:

Proxmox itself already has support for ZFS, which means you can setup a RAID from Proxmox directly, if you wish. Similarly, Proxmox can also expose the RAID's contents via Samba directly.

That sounds perfect! I do not have "a need" to use a NAS OS specifically, just the need to have the share available through common protocol(s). So I assume it goes something like this:

1. Install Proxmox on the M.2 SSD
2. Create a ZFS RAID with all the SATA HDD's
3. Enable Samba sharing for the ZFS RAID
4. Create a VM stored on the ZFS RAID
5. On that guest VM who needs to use the shared space; mount the ZFS RAID to the guest filesystem, and read/write to "smb://shared/nvr/camera0" for example

Does that sound like the right approach?

 

Also; if you mount that ZFS RAID to a local PC, will VM disk files and ISO files also just show up there in some folder? (like; "smb://vm" or "smb://iso")

[WereCatf]

10 minutes ago, Amadea said:

On that guest VM who needs to use the shared space; mount the ZFS RAID to the guest filesystem, and read/write to "smb://shared/nvr/camera0" for example

I'd rather recommend using NFS instead of Samba for mounting of any shared files to the VM, since NFS has far better performance, especially with small files and random reads/writes.

13 minutes ago, Amadea said:

Also; if you mount that ZFS RAID to a local PC, will VM disk files and ISO files also just show up there in some folder? (like; "smb://vm" or "smb://iso")

That entirely depends on whether you share those directories via Samba or not.

[Amadea]

Much thank for your feedback on the software side @WereCatf!

19 hours ago, WereCatf said:

I'd rather recommend using NFS instead of Samba for mounting of any shared files to the VM, since NFS has far better performance, especially with small files and random reads/writes.

Is this a build-in GUI feature of Proxmox, or would I have to configure the NFS or SMB package myself?

 

Any advise on hardware would be very welcome. I'm currently considering; 10500, 10600, 10600K, 10700, 10700K, 10850K with an ASUS ROG STRIX B460-F.

[WereCatf]

10 minutes ago, Amadea said:

Is this a build-in GUI feature of Proxmox, or would I have to configure the NFS or SMB package myself?

Sorry, I've only used Proxmox for like 5 minutes and I just can't remember every single thing about it. If you have any old PC lying around, you could always install Proxmox there and see all the different stuff you can do in it, or you can install Proxmox itself in a VM.

 

Also, now that I remember to mention it: you'd most likely get more Proxmox-specific help and advice on their own forums at https://forum.proxmox.com/ -- I run Qemu+KVM under plain Ubuntu and thus I just am not a good source for advice with Proxmox-specific stuff.

[Amadea]

(bump) Any advise on the hardware would be appreciated

[Amadea]

On 2/6/2021 at 4:27 PM, WereCatf said:

Intel's iGPU can be shared between multiple VMs, yes. I am using one such setup myself. That said, I don't know how to set such a system up in Proxmox. Google around for GVT-G or look at e.g. https://wiki.archlinux.org/index.php/Intel_GVT-g

@WereCatf, I am currently considering the Intel i7-10700 which should support VT-d and VT-x. And a Gigabyte H470 HD3 which supports VT-d according to the manual. Having checked that, can I be sure that passing AND sharing the iGPU (especially Intel Quick Sync Video) between VM's will work on this hardware? Strictly hardware speaking, not the configuration etc. that your link provided.

[Electronics Wizardy]

On 2/7/2021 at 4:05 AM, Amadea said:

Much thank for your feedback on the software side @WereCatf!

Is this a build-in GUI feature of Proxmox, or would I have to configure the NFS or SMB package myself?

 

Any advise on hardware would be very welcome. I'm currently considering; 10500, 10600, 10600K, 10700, 10700K, 10850K with an ASUS ROG STRIX B460-F.

You own't need nfs or samba at all, you can just put the vms on directly if you want. If you want shared storage, id setup a vm/container to make a smb/nfs share. You can do it on the host, I don't think proxmox has a gui for it, but all the samba/nfs tutorials will work fine.

 

1 hour ago, Amadea said:

@WereCatf, I am currently considering the Intel i7-10700 which should support VT-d and VT-x. And a Gigabyte H470 HD3 which supports VT-d according to the manual. Having checked that, can I be sure that passing AND sharing the iGPU (especially Intel Quick Sync Video) between VM's will work on this hardware? Strictly hardware speaking, not the configuration etc. that your link provided.

Yea that shoudl work. I don't think proxmox has a gui for it, so you have to edit some grub config files to tell it how to work, but it should work.

 

I can't garantee it will work as there are plenty of weird issues here, esp with consumer hardware that really isn't made for this use.

[WereCatf]

7 hours ago, Amadea said:

Having checked that, can I be sure that passing AND sharing the iGPU (especially Intel Quick Sync Video) between VM's will work on this hardware?

At least according to the relevant file in the source-code, the Comet Lake is supported by GVT-G and 10700 is a Comet Lake CPU.

 

That said, for optimal compatibility, just simply getting a 2nd GPU would be the way to go.

[Amadea]

On 2/15/2021 at 5:06 AM, WereCatf said:

That said, for optimal compatibility, just simply getting a 2nd GPU would be the way to go.

If it ends up not working or not performing well enough that might be an option. Although I am not familiar with hardware transcoding on a dedicated GPU. I believe that one of the services I want te run (TVheadend) doesn't even support transcoding with hardware acceleration.

 

On 2/14/2021 at 10:29 PM, Electronics Wizardy said:

You own't need nfs or samba at all, you can just put the vms on directly if you want. If you want shared storage, id setup a vm/container to make a smb/nfs share. You can do it on the host, I don't think proxmox has a gui for it, but all the samba/nfs tutorials will work fine.

So: create a ZFS RAID in Proxmox, assign VM disks to the ZFS but with minimal disk size, create a VM disk to cover the majority of the available ZFS space, and then create a file sharing service on that VM. Do I understand your approach correctly?

 

On 2/14/2021 at 10:29 PM, Electronics Wizardy said:

I can't garantee it will work as there are plenty of weird issues here, esp with consumer hardware that really isn't made for this use.

Yeah, seems that at this stage the only thing I can do is order and try it out. Nothing more I currently can lookup to check if things will work I think.

[WereCatf]

1 minute ago, Amadea said:

If it ends up not working or not performing well enough that might be an option. Although I am not familiar with hardware transcoding on a dedicated GPU. I believe that one of the services I want te run (TVheadend) doesn't even support transcoding with hardware acceleration.

I have no idea if TVheadend does support such or not. Speaking of which, Plex does support hardware-accelerated transcoding, but doesn't support TVheadend as Plex uses its own system for live-TV/DVR-functionality. I'm just mentioning this since you mentioned Plex in your first post.

[Electronics Wizardy]

5 hours ago, Amadea said:

So: create a ZFS RAID in Proxmox, assign VM disks to the ZFS but with minimal disk size, create a VM disk to cover the majority of the available ZFS space, and then create a file sharing service on that VM. Do I understand your approach correctly?

 

On 2/14/2021 at 1:29 PM, Electronics Wizardy said:

Yup thats what ID do