Running ESXi (VMVisor) 7.0U3f as a virtual machine on TrueNAS SCALE 22.02.2.1
So,
I tried a few things out, starting with Proxmox containing an ESXi 7 and TrueNAS Core VM, then realizing that my internal network needs a "router" of sorts (DHCP, gateway, etc) and experimenting between vyOS, RouterOS (Microtik) and pfSense on said Proxmox instance, totaling the number of virtual machines to 3.
I learned a few things and also didn't sleep for two nights but here's what I learned so far, I'm writing this down from memory so I could be completely wrong, please correct me. Everything specified is as of this writing.
-
PCI pass-through on Proxmox is hit-or-miss and is considered experimental for a reason
It'll still work, just... breakage is expected
-
If you have created a pool and configuration on TrueNAS Scale, you cannot just install TrueNAS Core and restore it as it has a lower version number
This is probably by design, apparently SCALE uses a greater version of ZFS than Core
-
Just because something is Linux-based doesn't mean it can be made into an LXC package
I found myself doing this while I was on Proxmox and "optimizing" TrueNAS SCALE and vyOS by converting their clean installs into LXC container tarballs despite me getting neither working yet and having never used LXCs before
-
RouterOS is paid software
I didn't read deep enough until I got it working and realized that it needs a license, in my defense, I was sleep deprived
-
Use pfSense
Yeah....
-
You can't mix nested virtualization and PCI passthrough with ESXi
See https://kb.vmware.com/s/article/67272
On 8/5/2022 at 1:04 AM, LIGISTX said:No need for new hardware... Just need to backup your truenas XML, format your boot media with ESXi, use an HBA to pass the drives through (thats 50 bucks.....), instal a truenas VM, load XML, done. Truenas will work just as it always did, it will just be running under ESXi.
On 8/5/2022 at 12:58 AM, Electronics Wizardy said:You probably can install esxi on the host, then pass the disk to a truenas vm, then use that vm to store the other vms on.
I essentially did this. Proxmox wasn't doing it for me, it felt jank to have a hypervisor within a hypervisor and if I had to use Proxmox, why use ESXi but I don't like Proxmox so I gave ESXi a shot.
I was avoiding it for so long because I thought it won't be compatible with my hardware and surprisingly.... it is!
While I was on Proxmox, after battling both vyOS and RouterOS, I found myself using pfSense (which is an amazing piece of software) and using the experience I gained from trying to run ESXi+SCALE+pfSense under Proxmox, I decided to ditch Proxmox and go pfSense+SCALE on ESXi...
It took a lot of research, false starts and failed attempts but finally, I...
-
Installed ESXi
I needed to use the flag systemMediaSize=min to discourage ESXi to allocate too much space to the OS (VMFSL) and leave some for the local store (VMFS)
-
Used the free space to create a new partition and assign it as a local store
This step really stumped me since Proxmox gives you a local store but ESXi 7 requires me to go and enable SSH so I can go and create datastore following this guide on Github Gist and until I figured it out, ESXi felt like an intimidating ghost town
-
Whitelisted my SATA controller
I found a guide that guided me to ensure that my controller won't be greyed out when I try to enable PCIe passthrough by modifying passthru.map
-
Went into maintenance mode and set the SATA controller as passthrough enabled and then rebooted the ESXi Host, then disabled SSH and exited maintenance mode
-
Upload my pfSense and SCALE ISOs, create a SCALE VM and assign the passed through SATA controller to the SCALE VM
Turns out, when you assign RAM to a VM, the RAM isn't "hard allocated" to that VM but is rather "soft allocated" (made available when needed) by default and mixing that with passthrough makes ESXi upset with this error
QuoteInvalid memory setting for FPT: memory reservation (sched.mem.min) should be equal to memsize(xxxx)
The solution is to go into advanced options within the VM's settings and ensure that the memory assigned equals the memory reserved.
-
Create 4 port groups, one for management, one for ISCSI, one for accessing the internet (through my general home network) and one for the intranet (managed by the pfSense instance inside)
I named them ManageNet (bound to statically assigned IP attached to my home network through VMkernel NIC 0), PublicNet (used by all VMs as vNIC0), PrivateNet (used by all VMs as vNIC1) and SCSINet (bound to statically assigned IP attached to the internal network through VMkernel NIC 1)
P.S. PublicNet is just the standard "VM Network" port group renamed
-
Assign two NICs to each VM
-
Start them up, install what's needed, import configurations and keys
-
Setup iSCSI to connect to the SCALE instance and use the exposed drive as the ZFS-backed data store for non-core VMs
The way I'm doing it is jank so I won't be elaborating any further on it :P... I didn't expect this to work at all but it somehow did!
-
Have the logs set up on persistent storage by following the steps of a knowledge base article
and...
.
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now