Storage on / between VMs

[xXDeltaXx]

Hi,
Bit of a noob on Proxmox / VMs… I’ve been following a few tutorials and am trying to work out the best way to organise the storage between VMs on a server…
It’s a Proxmox install across 2x 1TB NVME drives
I have a TrueNAS Scale VM which is intended to be for mass storage, backups, etc.
I have a Windows Server 2022 VM which will be mainly for Solidworks PDM, but also Active Directory and maybe a few other bits and bobs.
There will likely be other VMs for plex, maybe a couple of other things(?)
The VMs have their install on the Proxmox drives, and I am wondering how best to divide up / pass through the other storage.

Currently I have 4x 2TB NVME that I have tried creating as a ZFS in Proxmox and passing through to TrueNAS - it registered, but TrueNAS couldn’t get drive information, so I have redone by passing through as PCI Devices, and they register and can be Pooled just fine.

My current Windows server has 2x 2TB HDDs for the storage drives, so I only need about that on this new server.

I have another 4x 2TB NVME drives which are ready to setup, and a few temporary large SATA drives for bulk storage in TrueNAS (to be replaced in a little bit).

For TrueNAS, do I just continue to pass through the drives as a PCI Device and handle within the VM?

For Windows Server, do I do the same using Storage Spaces… or create a ZFS in Proxmox. . . or create something in TrueNAS and then somehow pass that over…?

Or is it all wrong and there’s a more elegant way for it all

[Electronics Wizardy]

3 hours ago, xXDeltaXx said:

but TrueNAS couldn’t get drive information, so I have redone by passing through as PCI Devices, and they register and can be Pooled just fine.

I'd setup the raid with zfs on proxmox if it was me. You can share in Truens if you wnt still with. virtual disk.

 

3 hours ago, xXDeltaXx said:

For Windows Server, do I do the same using Storage Spaces… or create a ZFS in Proxmox. . . or create something in TrueNAS and then somehow pass that over…?

I'd make a zfs disk in proxmox and then make a vitual disk for the vm.

 

Using virtual disks in stead of passthrough makes some things easier like snapshots and backups.

 

You can also make a network share in truenas/windows if you want the vms to sharestorage.

[LIGISTX]

Something to remember is VM’s “are just computers”. So if you want to “pass” storage from TrueNAS to windows server VM’s, you do it just like you would any normal PC; network shares. Being virtual, you get the benefit of virtio drivers…. Which are 10gb+ so virtual network is extremely fast. 

[xXDeltaXx]

As I understood it / understand it, Proxmox is the 'better' hypervisor and TrueNAS Scale is 'better' for storage... hence using both.

 

The logic for passing through the first set of drives to TrueNAS was based on initially creating the ZFS in proxmox and then TrueNAS highlighting that it had no drive SMART info. Researching more suggested individually passing them through as bare metal as 'better' for TrueNAS to handle them properly in a TrueNAS ZFS pool.
(I'm off to work, add more / another post when I'm back)

[LIGISTX]

29 minutes ago, xXDeltaXx said:

As I understood it / understand it, Proxmox is the 'better' hypervisor and TrueNAS Scale is 'better' for storage... hence using both.

 

The logic for passing through the first set of drives to TrueNAS was based on initially creating the ZFS in proxmox and then TrueNAS highlighting that it had no drive SMART info. Researching more suggested individually passing them through as bare metal as 'better' for TrueNAS to handle them properly in a TrueNAS ZFS pool.
(I'm off to work, add more / another post when I'm back)

I agree, and this is how I do it. I pass my drives through to TrueNAS via PCIe passthrough, for my HBA. 
 

But what exactly do you want to do? You can skin this cat via letting TrueNAS handle all storage, and could use iSCSI to present that to the windows machine, or you can let Proxmox handle it, and pass datasets (I believe you pad through datasets, zvols? I am actually not sure, never done it) through to VM’s. I think you need to figure out what you want your architecture to look like, both options, while technically different and which will require different management, are both totally valid options. 

[Electronics Wizardy]

10 hours ago, xXDeltaXx said:

As I understood it / understand it, Proxmox is the 'better' hypervisor and TrueNAS Scale is 'better' for storage... hence using both.

 

The logic for passing through the first set of drives to TrueNAS was based on initially creating the ZFS in proxmox and then TrueNAS highlighting that it had no drive SMART info. Researching more suggested individually passing them through as bare metal as 'better' for TrueNAS to handle them properly in a TrueNAS ZFS pool.
(I'm off to work, add more / another post when I'm back)

Well they both use ZFS for storage, so I'd say they do it equally well. 

 

Truenas has a better gui for network shares if you want that though.

 

You don't get smart data if you give truenas a virtual disk, but that isn't a issue as Proxmox can handle smart data along with all the other disks on the host.

[xXDeltaXx]

10 hours ago, LIGISTX said:

I agree, and this is how I do it. I pass my drives through to TrueNAS via PCIe passthrough, for my HBA. 
 

But what exactly do you want to do? You can skin this cat via letting TrueNAS handle all storage, and could use iSCSI to present that to the windows machine, or you can let Proxmox handle it, and pass datasets (I believe you pad through datasets, zvols? I am actually not sure, never done it) through to VM’s. I think you need to figure out what you want your architecture to look like, both options, while technically different and which will require different management, are both totally valid options. 

That's kind of where the learning / noob aspect comes into it. I know the general end result, but the way to get there is based off the best knowledge I have at that point in time.

I've got 2x1TB NVME drives to use as the main Prox mox / VM drives.

I've got an additional 8x 2TB NVME drives to have as fast storage for the Windows Server storage (2TB to match my current setup), and the rest to use as fast storage for TrueNAS/Plex/whatever gets added as I continue to build out.
There will be a few HDDs added either via an HBA or the motherboard SATA ports - just for mass storage and backups.

 

P1 is to replace my current Windows Server 2016 build with a Windows Server 2022 VM. This will be used for Solidworks PDM, with that Active Directory, and possibly some other Windows bits and bobs. (This then frees up that box to be rebuild as a backup server to back up the new one.

P2 is to add better network storage and proper back ups for my connected PCs / Laptops.

P3 is to add other features like Plex, maybe a Steam cache... once the main fucntions are there, then I can start looking around at other fun things / things to learn.

[LIGISTX]

41 minutes ago, xXDeltaXx said:

I've got an additional 8x 2TB NVME drives to have as fast storage for the Windows Server storage

So you want these drives to “live in” the windows VM? Or do you want them to be attached via a network share? If you want them to live “inside” windows, pass them through to windows. If you want them to be networked, pass them to truenas, then you can either present them as SMB shares, or if you want that windows VM to have much faster access (but out that VM to have access), you can use iSCSI. If you don’t know at that is, look it up, this may be helpful.

 

45 minutes ago, xXDeltaXx said:

and the rest to use as fast storage for TrueNAS/Plex/whatever gets added as I continue to build out.

Sounds like only a portion of the 8 NVMe drives you want for windows. If so, you should pass all of this to truenas, and use iSCSI to allow windows block level access to a certain amount of space, and then share the rest via SMB network shares. Also…. You really, really do not need Plex or anything remotely like Plex to live on SSD, none the less NVMe. You are paying a massive price premium for 0 benefit. 
 

Hopefully this is helpful and helps you start looking in the right directions. 

[xXDeltaXx]

1 minute ago, LIGISTX said:

So you want these drives to “live in” the windows VM? Or do you want them to be attached via a network share? If you want them to live “inside” windows, pass them through to windows. If you want them to be networked, pass them to truenas, then you can either present them as SMB shares, or if you want that windows VM to have much faster access (but out that VM to have access), you can use iSCSI. If you don’t know at that is, look it up, this may be helpful.

 

Sounds like only a portion of the 8 NVMe drives you want for windows. If so, you should pass all of this to truenas, and use iSCSI to allow windows block level access to a certain amount of space, and then share the rest via SMB network shares. Also…. You really, really do not need Plex or anything remotely like Plex to live on SSD, none the less NVMe. You are paying a massive price premium for 0 benefit. 
 

Hopefully this is helpful and helps you start looking in the right directions. 

I'm going to be having the Solidworks guy come out and do a 'supported' install of the PDM environment - so that means I can have the full support structure as I go. . . from that point of view, my current Windows Server setup has the main OS drive and a dedicated pair of 2tb HDDs for the actual PDM vault. I'd like to replicate that as closely as possible as I know it works. The 2x 1TB NVME drives with Proxmox and the Server VM (100GB at present) will be great for the OS, so it's just creating that 'D-drive'

From your suggestion I think it is sounding like setting all 8x 2TB NVMEs drives up to all pass through to TrueNAS and then look at setting up 2TB of that via iSCSI for the Drive is the way to go.

Thanks