Create a Single Server node acting as a VM source and Storage Server

[iadityaharsh]

Budget (including currency): INR 400,000

Country: India

Games, programs or workloads that it will be used for: 

3x Windows 10 Pro VMs accessible by 3 different Monitors, keyboard/ mouse set,

Storage Server (Probably with TrueNAS Core / Scale)

Other details (existing parts lists, whether any peripherals are needed, what you're upgrading from, when you're going to buy, what resolution and refresh rate you want to play at, etc): 

Parts I already have:

1. AMD Epyc 7502

2. Supermicro H12SSL-I

3. Corsair RM1000X

4. 128GB Micron 3200MT/s ECC RDIMM

5. 3x Nvidia Quadro T600 4GB

6. Asus Hyper M.2 x16 Gen4 Card

7. 4x Samsung 980 Pro 1TB

8. 2x Samsung 980 Pro 2TB

9. 2x Samsung 860 Evo 250GB

10. 6x Seagate Exos 14TB

 

I've attached an image to show my requirements.

Can anyone suggest which OS will fulfill my requirements? For storage Server I will probably stick to TrueNAS Core / Scale.

Apart from PCIe passthrough for GPU how can passthrough Keyboard / Mouse set to a VM through the onboard USB connectors.

 

Edited July 11, 2022 by iadityaharsh

[Electronics Wizardy]

Do you need vms here? ID probably just run windows 10/11 and use multiseat software like aster to have each user have their own dekstop setup and devices. You can also have users share gpus with this setup if you want.

 

You can also run a file server on windows.

 

If you want a hypervisor, Im a fan of proxmox, but almost any hypervisor will work. here.

 

Since you have the hardware, Why not try setting it up and see what issues you run into?

 

But If you can, Id create multiple systems. More reliable, cheaper, and faster.

[LIGISTX]

37 minutes ago, iadityaharsh said:

Budget (including currency): INR 400,000

Country: India

Games, programs or workloads that it will be used for: 

3x Windows 10 Pro VMs accessible by 3 different Monitors, keyboard/ mouse set,

Storage Server (Probably with TrueNAS Core / Scale)

Other details (existing parts lists, whether any peripherals are needed, what you're upgrading from, when you're going to buy, what resolution and refresh rate you want to play at, etc): 

Parts I already have:

1. AMD Epyc 7502

2. Supermicro H12SSL-I

3. Corsair RM1000X

4. 128GB Micron 3200MT/s ECC RDIMM

5. 3x Nvidia Quadro T600 4GB

6. Asus Hyper M.2 x16 Gen4 Card

7. 4x Samsung 980 Pro 1TB

8. 2x Samsung 980 Pro 2TB

9. 2x Samsung 860 Evo 250GB

10. 6x Seagate Exos 14TB

 

I've attached an image to show my requirements.

Can anyone suggest which OS will fulfill my requirements? For storage Server I will probably stick to TrueNAS Core / Scale.

Apart from PCIe passthrough for GPU how can passthrough Keyboard / Mouse set to a VM through the onboard USB connectors.

 

Proxmox? I’d look into proxmox. You can pass through USB controllers or individual devices. 
 

You will need an HBA to pass through the to truenas. Look for a Dell H310 oe similar LSI based SAS card. Get SAS to SATA cables. Make sure the card is flashed to IT mode, or flash it yourself. You DONT want it doing anything RAID functionality. It needs to act as an HBA or ZFS will not be happy. 

[iadityaharsh]

9 hours ago, Electronics Wizardy said:

Do you need vms here? ID probably just run windows 10/11 and use multiseat software like aster to have each user have their own dekstop setup and devices. You can also have users share gpus with this setup if you want.

 

You can also run a file server on windows.

 

If you want a hypervisor, Im a fan of proxmox, but almost any hypervisor will work. here.

 

Since you have the hardware, Why not try setting it up and see what issues you run into?

 

But If you can, Id create multiple systems. More reliable, cheaper, and faster.

It's already a multi computer system with low specs but there's security risk such as copying data over USB and these computers are going to be used by part time workers so security is paramount.

I thought of going with Proxmox except that Proxmox isn't perfect specially if SATA drives are to be considered, for once I also considered TrueNAS Scale but only one major feature is missing, USB passthrough though that coming soon as suggested within the community. What about unraid, LTT team has used that many times

[iadityaharsh]

9 hours ago, LIGISTX said:

Proxmox? I’d look into proxmox. You can pass through USB controllers or individual devices. 
 

You will need an HBA to pass through the to truenas. Look for a Dell H310 oe similar LSI based SAS card. Get SAS to SATA cables. Make sure the card is flashed to IT mode, or flash it yourself. You DONT want it doing anything RAID functionality. It needs to act as an HBA or ZFS will not be happy. 

I forgot to mention I have Dell H330 flashed into IT mode to become HBA330, any experience in passing through SATA drives? As my motherboard has 8 slots and 2 NVMe ports.

[Electronics Wizardy]

10 minutes ago, iadityaharsh said:

It's already a multi computer system with low specs but there's security risk such as copying data over USB and these computers are going to be used by part time workers so security is paramount.

I thought of going with Proxmox except that Proxmox isn't perfect specially if SATA drives are to be considered, for once I also considered TrueNAS Scale but only one major feature is missing, USB passthrough though that coming soon as suggested within the community. What about unraid, LTT team has used that many times

You can solve the security issues with multiple computers, I don't see how having one computer would be any more secure. Probably less secure as you now have more parts and systems to secure. You can easily disable lots of usb copying features in windows if thats an issue.

 

What won't proxmox do with sata drives? Id go proxmox as its much more flexable with storage than unraid or truenas scale, and you can configure drive passthrough if you want.

 

8 minutes ago, iadityaharsh said:

I forgot to mention I have Dell H330 flashed into IT mode to become HBA330, any experience in passing through SATA drives? As my motherboard has 8 slots and 2 NVMe ports.

Id just have the zfs storage managed by the proxmox host. Then you don't have to use passthrough at all.

 

But proxmox will do sata passthrough just fine if thats an issue.

[LIGISTX]

13 hours ago, Electronics Wizardy said:

Id just have the zfs storage managed by the proxmox host. Then you don't have to use passthrough at all.

 

Personally, I still like using truenas for managing ZFS; its just nice for that since its literally build specifically for it. But... no reason Proxmox can't do it, so fair point.

 

14 hours ago, iadityaharsh said:

I thought of going with Proxmox except that Proxmox isn't perfect specially if SATA drives are to be considered

What do you mean exactly? Why does the type of drive matter at all for the hypervisor host? 

 

14 hours ago, iadityaharsh said:

As my motherboard has 8 slots and 2 NVMe ports.

If you have a HBA, why would you also want to pass through drives? I have done it on a backup system which ran proxmox and truenas SCALE just to test it out, it works... But the ideal and preferred way is to use a HBA...... which you already have. On a PC build of this scale and cost, don't pass through SATA ports to truenas, use SAS HBA's and if you need more drives, get a SAS expander, or two, they are cheap and work exactly as advertised. Also, why do you need NVMe to be passed through? I assume the NVMe drives will be used to boot the guests, so just have them be native bare metal in proxmox and instal guest OS's on there. 

 

What exactly is the point of this system? It would help us a lot if we better understood what exactly you are trying to do.

[iadityaharsh]

On 7/12/2022 at 11:51 AM, LIGISTX said:

What exactly is the point of this system? It would help us a lot if we better understood what exactly you are trying to do.

So, I'm making this system as the Core server for 2 Needs: VMs with individual KVM access and Storage Solution.

 

I've a TrueNAS Server Running but it just doesn't fit in anymore.

My office workspace has 8 Employees accessing the Server for files on daily basis. These Files are mostly 100,000 and more of small files like documents, spreadsheets and pdfs, etc.

I'm having 2 ZPools inside TrueNAS Core, Pool-1 is a MIRROR Pool of 2 Pairs of 2TB NVMe SSDs (4x SAMSUNG 980 PRO 2TB); & Pool-2 is a RAIDZ2 Pool with 6x HDDs (Seagate EXOS 14TB).

For Room of Expansion I've space for another set of 6 Drives which I can add in Future.

 

Now the Problem, I've used Proxmox but didn't quite the way it handles GPU Passthrough and HDD Passthrough, so I want to avoid it unless necessary.

Secondly, I'm quite comfortable with TrueNAS (been using it for over 3 years now) and TN Scale is the immediate option I find viable. It has the ZFS which I already use, Passing through GPU is very easy and VMs inside ZPool vDEV means I can utilize all core functions + the community is way versed with TrueNAS. The only problem I have to face is passing through USB Devices, as discussed earlier I'm trying to use the VMs as if it were a Desktop so passing through Keyboard and Mouse isn't that easy (I've had a talk with a few support engineers of IX Systems and they assure that USB Passthrough is coming to Scale but it won't happen until December 2022 January 2023

 

Solution I'm trying to seek,

1. If I were to use TrueNAS Scale (because honestly I will transfer these VMs after Testing USB Passthrough on Scale on later date) how can I passthrough individual USB ports to the VM without adding separate USB PCIe cards to each VM.

2. Any other Hypervisor (like VMware ESXi or XCP-NG) that can help me in creating 4 VMs (3x Windows 10 & 1x TrueNAS Core) with all passthrough functionality and doesn't require too much setup like adding stuff via CLI (because honestly even if I'm comfortable in doing that, I don't have the time to debug things every time a screw-up happens.

[LIGISTX]

6 hours ago, iadityaharsh said:

So, I'm making this system as the Core server for 2 Needs: VMs with individual KVM access and Storage Solution.

 

I've a TrueNAS Server Running but it just doesn't fit in anymore.

My office workspace has 8 Employees accessing the Server for files on daily basis. These Files are mostly 100,000 and more of small files like documents, spreadsheets and pdfs, etc.

I'm having 2 ZPools inside TrueNAS Core, Pool-1 is a MIRROR Pool of 2 Pairs of 2TB NVMe SSDs (4x SAMSUNG 980 PRO 2TB); & Pool-2 is a RAIDZ2 Pool with 6x HDDs (Seagate EXOS 14TB).

For Room of Expansion I've space for another set of 6 Drives which I can add in Future.

 

Now the Problem, I've used Proxmox but didn't quite the way it handles GPU Passthrough and HDD Passthrough, so I want to avoid it unless necessary.

Secondly, I'm quite comfortable with TrueNAS (been using it for over 3 years now) and TN Scale is the immediate option I find viable. It has the ZFS which I already use, Passing through GPU is very easy and VMs inside ZPool vDEV means I can utilize all core functions + the community is way versed with TrueNAS. The only problem I have to face is passing through USB Devices, as discussed earlier I'm trying to use the VMs as if it were a Desktop so passing through Keyboard and Mouse isn't that easy (I've had a talk with a few support engineers of IX Systems and they assure that USB Passthrough is coming to Scale but it won't happen until December 2022 January 2023

 

Solution I'm trying to seek,

1. If I were to use TrueNAS Scale (because honestly I will transfer these VMs after Testing USB Passthrough on Scale on later date) how can I passthrough individual USB ports to the VM without adding separate USB PCIe cards to each VM.

2. Any other Hypervisor (like VMware ESXi or XCP-NG) that can help me in creating 4 VMs (3x Windows 10 & 1x TrueNAS Core) with all passthrough functionality and doesn't require too much setup like adding stuff via CLI (because honestly even if I'm comfortable in doing that, I don't have the time to debug things every time a screw-up happens.

For a work deployment, I don’t think I would use scale yet… it’s not mature yet. I think in 5 years it  will be an amazing option, but right now it’s stable enough for some enthusiast use, I wouldn’t trust it with any sort of company server yet. One of my buddies is running it at home and has had a few issues, and he is only doing basic stuff. I have not switched to it in my homelab, I am waiting for it to mature further. 
 

If using proxmox, you shouldn’t be passing through SATA ports, you need an HBA. Then you pass the entire HBA PCIe device through to truenas. Passing through individual SATA ports is possible, but as you experienced, it’s not ideal. I have not tried to pass a GPU through to a VM, but I assume it’s similar… just pass it through via the webUI, no?

 

XC-PNG can do all of this as well, they all just do it a little different. 
 

What do you mean “VM’s inside ZPOOL Vdevs”? If you need proxmox VM’s to have access to data on the storage server, network attach them, either via SMB, NFS, or iSCSI. Yes, if you use Scale the VM’s would be able to mount the data, but virtual networking will accomplish the same goal; virtual NICs more or less work via transferring data via shuffling RAM from VM to VM, so your CPU speed is the limiting factor for virtual networking between VM’s (which really means your drives are the bottleneck, as they would be if you used Scale). 
 

I admit, this is somewhat over my abilities and knowledge as I am only a homelab user, but I do not believe anything I have shared to be inaccurate. If you need a hypervisor, use a host that is explicitly designed to be one. You use truenas as a file server because it’s very good at what it does. I would suggest find a hypervisor host that is very good at being a hypervisor… hopefully with a few years of polish Scale will be both, but it’s not there quite yet