Nested virtualisation?

[ianm_ozzy]

Has anyone experience with this?

 

My bigger server has proxmox as an OS. It now has truenas scale as one of the virtual machines.

Truenas scale as a hypervisor?  I have tried it and are not  a big fan.

Also I am not keen the truenas scale 'apps'.

 

On an old machine, I was testing and messing around. I could run truenas scale with a virtual machine inside. It was  a brief test.

 

Hardware passthrough is fairly broken on the server. Passing through the onboard sata controller to truenas jsut froze the machine. 

All the setting in the BIOS were turned on & correct setup in proxmox.

Truenas scale has direct attached disks with no access to smart error data. This is dangerous.

 

I want to run proxmox within truenas scale instead.

It will be a single drive, with the boot & virtual machines on it.

 

Proxmox is far better as hypervisor & much easier to backup & restore virtual machines & LXC containers.

There will be nothing spectacular running, a low spec windows ( accessed through remote desktop) & a few LXC containers.

I am more concerned abut stability that super efficient performance.

 

It is not intended for long term. I am looking into the low power multi NIC machines as showcased by STH (serve the home).

 

Is anyone doing this?

 

So is running proxmox inside truenas scale stable possible?

 

 

 

 

[Blue4130]

8 minutes ago, ianm_ozzy said:

Has anyone experience with this?

 

My bigger server has proxmox as an OS. It now has truenas scale as one of the virtual machines.

Truenas scale as a hypervisor?  I have tried it and are not  a big fan.

Also I am not keen the truenas scale 'apps'.

 

On an old machine, I was testing and messing around. I could run truenas scale with a virtual machine inside. It was  a brief test.

 

Hardware passthrough is fairly broken on the server. Passing through the onboard sata controller to truenas jsut froze the machine. 

All the setting in the BIOS were turned on & correct setup in proxmox.

Truenas scale has direct attached disks with no access to smart error data. This is dangerous.

 

I want to run proxmox within truenas scale instead.

It will be a single drive, with the boot & virtual machines on it.

 

Proxmox is far better as hypervisor & much easier to backup & restore virtual machines & LXC containers.

There will be nothing spectacular running, a low spec windows ( accessed through remote desktop) & a few LXC containers.

I am more concerned abut stability that super efficient performance.

 

It is not intended for long term. I am looking into the low power multi NIC machines as showcased by STH (serve the home).

 

Is anyone doing this?

 

So is running proxmox inside truenas scale stable possible?

 

 

 

 

You want to run Truenas and then proxmox inside and then more stuff inside proxmox? Why? Since you already have proxmox as the bas now, why not just use it as is and have more than one vm?

[leadeater]

Why not run TrueNAS Scale as the VM and buy a cheap SAS/SATA PCIe card on ebay to attached the disks to and pass that through to the TrueNAS VM? The first layer should be the better Hypervisor not the worse one 

[ianm_ozzy]

I did explain. Truenas core sucks as a hypervisor. I will only need to install one virtual machine, specifically proxmox.

 

So proxmox: Easy to backup & redeply virtual machines on other server. Also Windows 11 TPM emulation.

Truenas scale: none of these options.  

[ianm_ozzy]

6 minutes ago, leadeater said:

Why not run TrueNAS Scale as the VM and buy a cheap SAS/SATA PCIe card on ebay to attached the disks to and pass that through to the TrueNAS VM? The first layer should be the better Hypervisor not the worse one 

 I explained hardware passthough is not working. It is hit & miss. Also no free pcie ports on the machine.

It is a machine for entertainment related stuff  using mainly parts I have already -  storing games & media. I am not spending more more money on it.

 

I think I mentioned that truenas scale is seriouly lacking as a hypervisor, well that is in my opinion.

 

Also how would I pass through the intel optane drives, which I use for deduplication.

 

[leadeater]

1 minute ago, ianm_ozzy said:

I explained hardware passthough is not working

Passing through onboard SATA hardly ever works though, doesn't mean passthrough itself isn't working.

 

1 minute ago, ianm_ozzy said:

Also no free pcie ports on the machine.

Ah damn that's unfortunate.

 

2 minutes ago, ianm_ozzy said:

Also how would I pass through the intel optane drives, which I use for deduplication.

These are PCIe devices so would need to use PCIe passthrough.

 

Anyway TrueNAS Scale is KVM Hypervisor so yes it can be used in the way you want to.

[ianm_ozzy]

15 minutes ago, leadeater said:

Passing through onboard SATA hardly ever works though, doesn't mean passthrough itself isn't working.

 

Ah damn that's unfortunate.

 

These are PCIe devices so would need to use PCIe passthrough.

 

Anyway TrueNAS Scale is KVM Hypervisor so yes it can be used in the way you want to.

 

 

On the machine, PCI passthrough does not work at  with anything attached through the chipset. Only directly attached to the CPU.

One of the optane drives is in an adapter PCIE to NVMe.

 

Even If I got a HBA card, plugged into the main PCIE slot, could only pass through the HBA card & 1 of the two optane drives. The other would be on the adapter connected through the chipset.

 

I need truenas scale for deduplciation &  ISCSI shares mainly. Sadly there is little documentation in the way of doing this in proxmox. I expect it is possible with some highly specialized linux command line stuff, but I do not  know.

 

 

Also where is the windows 11 TPM emulation in truenas scale?

 

 

Has anyone got any useful information on nested virtualization?

 

 

 

 

 

 

[Electronics Wizardy]

3 minutes ago, ianm_ozzy said:

 

 

On the machine, PCI passthrough does not work at  with anything attached through the chipset. Only directly attached to the CPU.

One of the optane drives is in an adapter PCIE to NVMe.

 

Even If I got a HBA card, plugged into the main PCIE slot, could only pass through the HBA card & 1 of the two optane drives. The other would be on the adapter connected through the chipset.

 

I need truenas scale for deduplciation &  ISCSI shares mainly. Sadly there is little documentation in the way of doing this in proxmox. I expect it is possible with some highly specialized linux command line stuff, but I do not  know.

 

Has anyone got any useful information on nested virtualization?

 

 

 

 

 

 

Why not make proxmox the host, and let it handle storage with zfs. 

 

Then make a vm for truenas, and give it a virtual disk on the proxmox storage pool.  Then you can use the truenas gui.

 

Also you probably don't want to use dedup in truenas/zfs is pretty ram happy and can kill performance.

 

Could also just make a container with debian or anouther os you want, and setup the iscsi server.

[leadeater]

8 minutes ago, ianm_ozzy said:

Has anyone got any useful information on nested virtualization?

TrueNAS Scale is just KVM, yes it works fine. There isn't isn't much you need to do other than have a supported CPU (vt-x & vt-d) and enable nested option in KVM.

 

https://www.howtogeek.com/devops/how-to-enable-nested-kvm-virtualization/

[ianm_ozzy]

Just now, Electronics Wizardy said:

Why not make proxmox the host, and let it handle storage with zfs. 

 

Then make a vm for truenas, and give it a virtual disk on the proxmox storage pool.  Then you can use the truenas gui.

 

Also you probably don't want to use dedup in truenas/zfs is pretty ram happy and can kill performance.

 

Could also just make a container with debian or anouther os you want, and setup the iscsi server.

 

 

Deduplication  with  ISCSI are the  main reasons  I have  to truenas scale. It is for multiple ISCSI shares. (12TB multiple game drives)

Getting deduplication to work well is very tricky, but worked it out. There was command line sorcery required in truenas scale.

 

Maybe you need to see this:

 

 

Has anyone useful information on nested virtualization?

 

[leadeater]

8 minutes ago, Electronics Wizardy said:

Then make a vm for truenas, and give it a virtual disk on the proxmox storage pool.  Then you can use the truenas gui.

Personally given the situation that's what I would do, well mostly. I would just do it all with Promox (second well, I'd never use Promox but I digress lol) and cli.

[Electronics Wizardy]

2 minutes ago, ianm_ozzy said:

 

 

Deduplication  with  ISCSI are the  main reasons  I have  to truenas scale. It is for multiple ISCSI shares. (12TB multiple game drives)

Getting deduplication to work well is very tricky, but worked it out. There was command line sorcery required in truenas scale.

 

Maybe you need to see this:

 

 

Has anyone useful information on nested virtualization?

 

YOu can run the exact same dedup on proxmox using ZFS if you want.

 

Also google guides for iscsi on debian. All of that should work on proxmox.

[ianm_ozzy]

5 minutes ago, leadeater said:

TrueNAS Scale is just KVM, yes it works fine. There isn't isn't much you need to do other than have a supported CPU (vt-x & vt-d) and enable nested option in KVM.

 

https://www.howtogeek.com/devops/how-to-enable-nested-kvm-virtualization/

 

Thank you

[ianm_ozzy]

6 minutes ago, Electronics Wizardy said:

YOu can run the exact same dedup on proxmox using ZFS if you want.

 

Also google guides for iscsi on debian. All of that should work on proxmox.

 

It is probably true that the dedup setup could be ran on proxmox.

I could not find any documentation on it.

Also the question though -  is it a good idea?

I expect it will involve serious networking, linux command line & zfs knowledge.

 

In truenas scale, it means creating the zvols on in the command line with a 1MB block size. In the gui the maximum option is 128k. The info was provided through the truenas forums.

All else is just setup through truenas GUI.

 

When shared  through ISCSI,  on the windows machine, connect to the ISCSI, format with 1MB block size.   This can be super wasteful of space when many small files are involved, but the compression on truenas helps with that a lot it seems.  Also it vastly reduced the deduplciation data needed - by 64 x.

 

The default zvol block  size in truenas is 16k. I used 1MB.  So 64 x less deduplicaton data required. So it seems the 2 cheap 16GB intel optane drives I have do the job.

I will see how it works out.

 

[LIGISTX]

2 hours ago, Electronics Wizardy said:

Why not make proxmox the host, and let it handle storage with zfs. 

 

Then make a vm for truenas, and give it a virtual disk on the proxmox storage pool.  Then you can use the truenas gui.

 

Also you probably don't want to use dedup in truenas/zfs is pretty ram happy and can kill performance.

 

Could also just make a container with debian or anouther os you want, and setup the iscsi server.

Now this…. Is a great example of how this setup can make a lot of sense. 

[Nick7]

As few have said.

Use proxmox.

Setup ZFS there, it's easy even from command line, also same for dedup.

 

For iSCSI I use SCST, as for me it performs better than LIO.

You can easily edit config, and add disks as needed. It really ain't rocket science.

 

This way, you have proxmox with KVM as hypervisor, native ZFS, native iSCSI, no need to nested virtualizations or other questionable things.

 

Proxmox is based on debian... so it's easy to add stuff.

[ianm_ozzy]

Just now, Nick7 said:

As few have said.

Use proxmox.

Setup ZFS there, it's easy even from command line, also same for dedup.

 

For iSCSI I use SCST, as for me it performs better than LIO.

You can easily edit config, and add disks as needed. It really ain't rocket science.

 

This way, you have proxmox with KVM as hypervisor, native ZFS, native iSCSI, no need to nested virtualizations or other questionable things.

 

Proxmox is based on debian... so it's easy to add stuff.

 It would be nice to know exactly how this is done?

 

As I undersand how this needs to be done?

 

So setup  a zfs pool with the deduplication drives?

That I could not find out. The proxmox documentation was a bit vague.

 

Then

Setup a couple of zvols with dedup tured on.

I kind of know how to to that in truenas.

 

Then setup a virtual machine & assign the block share device to it.

Not that difficult I expect.

 

Setup an ISCSI share on linux Virtual machine.

There is no LXC turnkey container I can find for this.

 

It seems  major linux command line, zfs & networking knowledge is required.

That is not really me.

 

 

 

 

 

 

 

 

 

[ianm_ozzy]

1 hour ago, Nick7 said:

As few have said.

Use proxmox.

Setup ZFS there, it's easy even from command line, also same for dedup.

 

For iSCSI I use SCST, as for me it performs better than LIO.

You can easily edit config, and add disks as needed. It really ain't rocket science.

 

This way, you have proxmox with KVM as hypervisor, native ZFS, native iSCSI, no need to nested virtualizations or other questionable things.

 

Proxmox is based on debian... so it's easy to add stuff.

 

is there a precise guide somewhere?

[ianm_ozzy]

8 hours ago, leadeater said:

Personally given the situation that's what I would do, well mostly. I would just do it all with Promox (second well, I'd never use Promox but I digress lol) and cli.

That is a problem

Apparently an iscsi share  can only be a zvol created with it. Not a raw drive passed to it, or an actual drive if running on bare metal.

That was apparently in the truenas scale docs. I did think of somethiing similar.

If I am wrong, please let me know how to do it.

 

[ianm_ozzy]

I tried nested virtualization on an old machine, installing truenas, using an old hard drive as a pool, then installing proxmox. In proxmox installing windows.

It worked.

 

Unless the suggestions of doing it within proxmox have a precise guide somewhere, that is what I will be doing.