unraid + thunderbolt 3 Gaming + Server build

[thecodingart]

Budget (including currency): N/A

Country: United States

Games, programs or workloads that it will be used for: HTPC, Kodi, Steam, Plex, HomeBridge, Various other experiments

Other details (existing parts lists, whether any peripherals are needed, what you're upgrading from, when you're going to buy, what resolution and refresh rate you want to play at, etc): 
NVIDIA 3090 Ti Founders Edition, ASUS ProArt Z690 - Board, 1300 Watt Platinum Power supply, i9-12900KF, 2x OWC Flex 1U4 4-Bay Rackmount Thunderbolt Storage

I'm currently building a HTPC that I would also like to use as a gaming PC. I've purchased the above specs (debating on changing to the 12900K rather KF), and have been toiling on the idea of using unraid as the primary OS with VMs containing graphic card passthroughs for virtualization. Some older posts seem to indicate issues with 12th gen intel chips + unraid atm, and I'm unsure of the 3090 support for unraid driver wise either (or if I should/need an integrated graphics card at all). I'm also extremely curious on thunderbolt compatibility as I haven't come across to many individuals who have tried creating an unraid from thunderbolt mounted drives (something that has surprised me a bit). This thread seems to indicate compatibility, but I'd like to see someone's experience down this road as I realize Linux + Thunderbolt has a bit of a rocky history: https://forums.unraid.net/topic/123906-list-of-working-thunderbolt-services-and-supported-thunderbolt-card-host-combinations-in-unraid/.

Any experience in this area would be helpful. I honestly wanted a modular and customizable rig, which is why I went this route rather trying to use a high end QNAP (also didn't want to deal with the overall limitations of using a hypervisor QNAP station as I've read they can lead to nasty situations). I'm not even 100% sure if there's a clean way to add a 3090 Ti to a QNAP server either.

 

[Electronics Wizardy]

If you want to do gpu passthrough, having the igpu on the cpu will be very nice, and Really get the non f version

 

Why the 1u 4 bay storage boxes? Id get rid of those and get a sas jbod or put the drives in the case, or just go usb. Thunderbolt just doesn't make sense here.

 

Id normally recommend keeping the gaming pc separate from the server. Makes it easier to manage and work with, and often cheaper.

 

Or just run windows on the host. Then the gaming won't have any issues, and things like hotplug and thunderbolt work well, and run vms in hyper-v as needed.

 

[FloRolf]

Honestly I'd stay away from all of that. 

1. Running a 12900k and 3090 24/7 to keep the server running will drive electricity usage and cost way up

2. Troubleshooting will be more difficult 

3. Why thunderbolt in the first place? Do you want to use SSDs? Why not just sata? 

 

-> get a synology or maybe even a TrueNAS and a separate gaming pc. 

[thecodingart]

21 minutes ago, FloRolf said:

Honestly I'd stay away from all of that. 

1. Running a 12900k and 3090 24/7 to keep the server running will drive electricity usage and cost way up

2. Troubleshooting will be more difficult 

3. Why thunderbolt in the first place? Do you want to use SSDs? Why not just sata? 

 

-> get a synology or maybe even a TrueNAS and a separate gaming pc. 

Having this as an all in one is more important to me and I’m more interested in using unraid to help manage some of that power usage. I’m not interested in using SSDs for the Plex portion, and these particular bays are compatible with SSDs if I was. The main board will have some 980 SSDs in it for the gaming portion and main system.

[thecodingart]

47 minutes ago, Electronics Wizardy said:

If you want to do gpu passthrough, having the igpu on the cpu will be very nice, and Really get the non f version

 

Why the 1u 4 bay storage boxes? Id get rid of those and get a sas jbod or put the drives in the case, or just go usb. Thunderbolt just doesn't make sense here.

 

Id normally recommend keeping the gaming pc separate from the server. Makes it easier to manage and work with, and often cheaper.

 

Or just run windows on the host. Then the gaming won't have any issues, and things like hotplug and thunderbolt work well, and run vms in hyper-v as needed.

 

What do you happen to mean when noting to run Windows on the host? I’m definitely planning on using the PC drives and host computer to run Windows, but I was thinking about using unraid and passing through the GPU while giving unraid the integrated graphics card. I’m honestly unaware if you can accomplish the same thing in Windows 11 Pro where you can split/dedicate a GPU so it’s not active unless the VM is active (this would be perfect).

 

Thunderbolt is a desire of mine from a flexibility perspective (speed, daisy chaining, and multipurpose plug n play). Overall, it plays into the majority of my home network and rig in a more modular fashion at a PCI like speed (something I appreciate). It’s far more useful at scale other than just using it as a storage bus. 
 

I have considered miniSAS, but it has its own high cost buy in and is less multi purpose (and barely carries anymore bandwidth in comparison to Thunderbolt).

 

As another note, per having a modular system, I do want to provide max bandwidth on any system (including Apple based computers and devices).

[manikyath]

i daily drive unraid with a virtual desktop with GPU passtrough to run some stuff i want to carry between my desktop and laptop, so that the desktop doesnt have to keep running.

 

it's *not* a nice experience, if you arent playing by the rules.

- modularity.. isnt a thing. your pcie passtrough configs will end up all confused each time you add or remove something.

- to add or remove hardware, to troubleshoot your nvidia drivers (because you will be), and when your VM's OS decides sto do weird things.. you're rebooting the entire party, manually shutting down each VM, and manually spooling things up again.

- it's *not* something that'll save power, quite the opposite in fact.

- storage... is slow. not "painfully slow", but your main gaming VM is only gonna get a fraction of the IOPS it would get running bare metal.

- unraid is *NOT* built for removable storage, thunderbolt is a wasted exercise because anything part of your array must stay connected, and stay connected in mostly the exact same position.

 

at the very least, separate out the gaming rig. if you insist on having everything in one place, stack two cases on top of each other, or perhaps.. put the gaming pc at your desk, and the HTPC near your TV, like you'd expect things to be.

[thecodingart]

17 minutes ago, manikyath said:

i daily drive unraid with a virtual desktop with GPU passtrough to run some stuff i want to carry between my desktop and laptop, so that the desktop doesnt have to keep running.

 

it's *not* a nice experience, if you arent playing by the rules.

- modularity.. isnt a thing. your pcie passtrough configs will end up all confused each time you add or remove something.

- to add or remove hardware, to troubleshoot your nvidia drivers (because you will be), and when your VM's OS decides sto do weird things.. you're rebooting the entire party, manually shutting down each VM, and manually spooling things up again.

- it's *not* something that'll save power, quite the opposite in fact.

- storage... is slow. not "painfully slow", but your main gaming VM is only gonna get a fraction of the IOPS it would get running bare metal.

- unraid is *NOT* built for removable storage, thunderbolt is a wasted exercise because anything part of your array must stay connected, and stay connected in mostly the exact same position.

 

at the very least, separate out the gaming rig. if you insist on having everything in one place, stack two cases on top of each other, or perhaps.. put the gaming pc at your desk, and the HTPC near your TV, like you'd expect things to be.

If I were to use unraid or hyper-v and dedicated the integrated graphics card for plex + the main machine, while the 3090 Ti was dedicated to the gaming portion, wouldn't that save a significant amount of power as the 3090 Ti would literally shut down with the gaming VM?

Also, regarding thunderbolt, I'm not sure I understand your comment there as the speed is not just for/from the I/O on the drives (which is theoretically maxed out at this point). It's also for daisy chaining any bandwidth I'd like ... which could include an external GPU enclosure or something else. When referring to "modularity" I'm simply speaking from a daisy chaining, upgradability, and some device agnostics perspective. For example, if I desired to hook up a Mac mini to these drives, that would be easiest to accomplish through a USB-C/Thunderbolt 3/4 port rather anything tied to mini-SAS. Small example, but really want to hammer on the point that mini-SAS is not nearly as multipurpose or portable.

Please do keep in mind the external drives are not for gaming, they are for other partitions. But the advice of order of operations on unraid is an interesting comment.

You're experiences with unraid do make me more curious about a hyper-v approach with a standard SoftRaid setup.

As another side-note, I do have a Mac Studio that I could use to run as a primary machine for work purposes + my non-gaming setup. Having the Thunderbolt setup makes that migration fairly easy in all honesty, but I was very tempted to go down the route of using a Virtual machine with the gaming PC instead.

[Kilrah]

48 minutes ago, thecodingart said:

while the 3090 Ti was dedicated to the gaming portion, wouldn't that save a significant amount of power as the 3090 Ti would literally shut down with the gaming VM?

Don't know about a 3090Ti, but a 3080 will draw 50W as long as it's connected and powered. There is no software "shutdown", it'll still be running, just doing nothing.

 

On Unraid (and TBF would be valid for others too) external array storage is highly discouraged. One software/hardware hiccup that causes a temporary disconnect and you're in for some "fun". 

[thecodingart]

1 hour ago, Kilrah said:

Don't know about a 3090Ti, but a 3080 will draw 50W as long as it's connected and powered. There is no software "shutdown", it'll still be running, just doing nothing.

 

On Unraid (and TBF would be valid for others too) external array storage is highly discouraged. One software/hardware hiccup that causes a temporary disconnect and you're in for some "fun". 

Interesting, although I fail to see the difference in this recommendation vs any external mounting + Raid tool like SoftRaid? I’ve personally been using SoftRaid with Thunderbolt for years and it’s been as rock solid as a SAS port is. Maybe the recommendation comes from unraid not really being a raid for striped? I’m also guessing that’s a separate note than using hyper-v. eGPUs are a thing for a reason via thunderbolt

 

The power draw notes bite though. Definitely not what I was hoping to hear.

[manikyath]

7 hours ago, thecodingart said:

If I were to use unraid or hyper-v and dedicated the integrated graphics card for plex + the main machine, while the 3090 Ti was dedicated to the gaming portion, wouldn't that save a significant amount of power as the 3090 Ti would literally shut down with the gaming VM?

no, because the card is still powered, and running idle.

7 hours ago, thecodingart said:

which could include an external GPU enclosure or something else

that sounds like a bad idea with unraid.

7 hours ago, thecodingart said:

For example, if I desired to hook up a Mac mini to these drives, that would be easiest to accomplish through a USB-C/Thunderbolt 3/4 port rather anything tied to mini-SAS.

if you want to use the drivers with unraid, they have to be in a format that unraid understands, and by extension a mac will most likely not understand. also: disconnecting drives from unraid is a great way to have a bad time. use SMB or FTP if you want to access the drives from another system.

7 hours ago, thecodingart said:

You're experiences with unraid do make me more curious about a hyper-v approach with a standard SoftRaid setup.

even less suitable for what you plan to be doing.

 

7 hours ago, thecodingart said:

As another side-note, I do have a Mac Studio that I could use to run as a primary machine for work purposes + my non-gaming setup. Having the Thunderbolt setup makes that migration fairly easy in all honesty, but I was very tempted to go down the route of using a Virtual machine with the gaming PC instead.

map the storage as a network drive. that's how unraid is supposed to work.

[thecodingart]

4 hours ago, manikyath said:

no, because the card is still powered, and running idle.

that sounds like a bad idea with unraid.

if you want to use the drivers with unraid, they have to be in a format that unraid understands, and by extension a mac will most likely not understand. also: disconnecting drives from unraid is a great way to have a bad time. use SMB or FTP if you want to access the drives from another system.

even less suitable for what you plan to be doing.

 

map the storage as a network drive. that's how unraid is supposed to work.

Quote

map the storage as a network drive. that's how unraid is supposed to work.
 

Yeah, I have absolutely 0 interest in doing this. Network mounted drives are frankly terrible in the way OSs handle them. I’m still not 100% certain why something like hyper-v (which is just an image pointing to something’s) is less suited for my use case considering I would just give it a drive mount to access. I have no interest is having a drive mounted as a network or SCSI though. I guess I’ll have to experiment with this on the side. The idle power and the way unraid bridges drives lacking portability is probably my main learnings here.