The Pure Solid State Server Build Log

[Windows7ge]

10 hours ago, maxtch said:

Since you have that much compute power, you can replace other PC's in the room with Raspberry Pi's acting as thin clients.

 

Meanwhile if you have the funds, you may want to scoop up some of those 4TB Intel DC P4000 NVMe SSD's Facebook was dumping, and a maybe one of those nVidia Tesla cards that can be modded into a GRID. The former are just some cheap and fast SSD, and the latter will allow you assign virtual GPU to the VM's. 

The original design of the server was going to include 8 NVMe compatible drive bays but I opted not to go that route. Of course I could always change my mind in the future. For the time being growing the SATA SSD pool should still result in a ludicrous number of IOPS and throughput. More than I can push though a couple aggregated 10Gbit NICs.

 

My other weaker server already partially plays the role of hosting a VM with GPU acceleration. I can't fully move away from Windows but I can host it in a VM. Give it a GPU and remote in with something like PARSEC. I am curious to explore GPU's that allows their resources to be divided though processes such as SR-IOV. I just recently picked up a hot air rework station. I wouldn't mind tinkering with vGPU technology.

[leadeater]

10 hours ago, Windows7ge said:

I am curious to explore GPU's that allows their resources to be divided though processes such as SR-IOV

As long as you're prepared to dump buckets of cash at not very high performance GPUs. SR-IOV support GPUs are stupid expensive even used so you'll end up paying a lot for slow or bloody heaps for fast. Wouldn't mind a SR-IOV GPU myself but... screw that price tag lol.

[Windows7ge]

22 minutes ago, leadeater said:

As long as you're prepared to dump buckets of cash at not very high performance GPUs. SR-IOV support GPUs are stupid expensive even used so you'll end up paying a lot for slow or bloody heaps for fast. Wouldn't mind a SR-IOV GPU myself but... screw that price tag lol.

I don't know much about the field which is why I'd like to play with them. If I'm not mistaken SR-IOV is not a AMD specific feature but some form of open standard as I've heard about it's use for dividing ports on NICs to multiple VMs where you can't pass-through the ports became they all show up in the same IOMMU group.

 

I believe NVIDIAs implementation they just call vGPU. Modding old NVIDIA cards seem to be a cheap way to go to get something with more features.

 

I'll have to look into it more. Right now though I won't need the VMs to have any GPU acceleration. I just need them to act as clients on a network. This won't happen for some time though. I have a bunch of little projects I need to work though first.

[leadeater]

6 minutes ago, Windows7ge said:

I don't know much about the field which is why I'd like to play with them. If I'm not mistaken SR-IOV is not a AMD specific feature

Correct but when it comes to GPUs only AMD has it, Nvidia uses custom software drivers to achieve the same result. When it comes to using Nvidia's vGPU solution I believe it also requires licenses, something additional to say VMware Horizon licenses but I could be wrong on that. I should know since we have both Horizon and Nvidia GPUs in the VDI cluster but the servers come with everything required so never really checked the bill of materials.

 

So yea SR-IOV is exclusive to AMD for GPUs and Nvidia are dicks so make sure you'll be able to use what you buy for the purpose you buy it for.

[Windows7ge]

1 hour ago, leadeater said:

Nvidia are dicks so make sure you'll be able to use what you buy for the purpose you buy it for.

Nvidia doesn't let you pass their desktop series cards though to VMs. The GPUs check if they're running in a VM. When you install the display driver it will disable itself and Device Manager reports Error 43. So they're double dicks.

 

AMD on the other hand is like, do whatever you want. The last time I paid attention to AMD's Workstation cards was the Firepro series and that was years ago. I'd have to find a software that supports the SR-IOV function. It's possible I could do it directly though QEMU by editing the .XML file but I would have no idea what to type in.

 

For now it doesn't matter though. I don't plan for the VMs to need or use GPU acceleration. Rendering the desktop off the CPU will get the job done for what I plan to do.

[maxtch]

On 1/1/2020 at 12:07 PM, Windows7ge said:

Nvidia doesn't let you pass their desktop series cards though to VMs. The GPUs check if they're running in a VM. When you install the display driver it will disable itself and Device Manager reports Error 43. So they're double dicks.

 

AMD on the other hand is like, do whatever you want. The last time I paid attention to AMD's Workstation cards was the Firepro series and that was years ago. I'd have to find a software that supports the SR-IOV function. It's possible I could do it directly though QEMU by editing the .XML file but I would have no idea what to type in.

 

For now it doesn't matter though. I don't plan for the VMs to need or use GPU acceleration. Rendering the desktop off the CPU will get the job done for what I plan to do.

If you want to put nVidia GPU in VM's I think the only cheap solution is hacking those Teslas into GRID's and pass them through using vGPU. As of passing AMD GPU, those mining RX 470's flashed with gaming VBIOS passed over SR-IOV are good options since those cards are cheap and lacked video output.

 

Anyway, after it is done you can just put the server somewhere out of earshot, and use a bunch of Pi's in the rooms as thin clients.

[Windows7ge]

On 12/31/2019 at 9:12 PM, leadeater said:

As long as you're prepared to dump buckets of cash at not very high performance GPUs. SR-IOV support GPUs are stupid expensive even used so you'll end up paying a lot for slow or bloody heaps for fast. Wouldn't mind a SR-IOV GPU myself but... screw that price tag lol.

By a strange twist of coincidence someone here recently bought a AMD GPU workstation series card. The AMD Radeon Pro WX2100. The specs are not impressive but it's based on their MxGPU line-up and supports SR-IOV letting you run up to 16 simultaneous GPU accelerated virtual machines. All for the pretty low cost of ~$125. That's cheap enough to where I'd be willing to buy it just to play with SR-IOV.

 

There were demonstrations of it being used with VMware ESXi. Any idea if there are free licensees for that? If not I'l have to dive deep into the Debian/QEMU documentation to find out if it's possible to assign one of the 16 available vGPUs. It's going to be a pain because I've been told that SR-IOV is meant to be a solution to getting around IOMMU groups.

[leadeater]

2 hours ago, Windows7ge said:

There were demonstrations of it being used with VMware ESXi. Any idea if there are free licensees for that?

The basic free version of ESXi should suffice. I've used SR-IOV NICs with it, you enable it then select how many NICs to generate so I'd assume it's similar for a GPU with SR-IOV, requires reboot of host btw.

 

Then you just edit a VM and add a PCIe device and it'll list all the SR-IOV NICs or GPU (probably).

[maxtch]

On 1/26/2020 at 6:53 AM, Windows7ge said:

By a strange twist of coincidence someone here recently bought a AMD GPU workstation series card. The AMD Radeon Pro WX2100. The specs are not impressive but it's based on their MxGPU line-up and supports SR-IOV letting you run up to 16 simultaneous GPU accelerated virtual machines. All for the pretty low cost of ~$125. That's cheap enough to where I'd be willing to buy it just to play with SR-IOV.

 

There were demonstrations of it being used with VMware ESXi. Any idea if there are free licensees for that? If not I'l have to dive deep into the Debian/QEMU documentation to find out if it's possible to assign one of the 16 available vGPUs. It's going to be a pain because I've been told that SR-IOV is meant to be a solution to getting around IOMMU groups.

That card is RX 540 with PCIe x8 interface...

[Windows7ge]

1 minute ago, maxtch said:

That card is RX 540 with PCIe x8 interface...

...and what you're getting at is...

[maxtch]

3 minutes ago, Windows7ge said:

...and what you're getting at is...

Miners' RX 470 are likely better deals if you are just SR-IOV'ing.

[Windows7ge]

Just now, maxtch said:

Miners' RX 470 are likely better deals if you are just SR-IOV'ing.

RX470 has SR-IOV enabled? I was told AMD never enabled the feature for any of their desktop series cards.

 

If it's cheaper and lets me create a handful of vGPU's I'm open to going that direction.

[maxtch]

2 minutes ago, Windows7ge said:

RX470 has SR-IOV enabled? I was told AMD never enabled the feature for any of their desktop series cards.

 

If it's cheaper and lets me create a handful of vGPU's I'm open to going that direction.

My mistake. It would be more bang for the buck if you pass through miners' RX 470's instead of SR-IOV'ing what is effectively an RX 540.

[Windows7ge]

4 minutes ago, maxtch said:

My mistake. It would be more bang for the buck if you pass through miners' RX 470's instead of SR-IOV'ing what is effectively an RX 540.

According to my understanding GPU passthough ≠ SR-IOV. The functions are different. Both have the same end result for the VM a graphics accelerator but the setup is completely different. I want to explore having one GPU host multiple VMs.

 

Getting Leadeater's attention again I've done a little digging in the PROXMOX documentation and it seems there a chance QEMU supports vGPU. I may not need VMWare ESXi to play with SR-IOV but there's still a chance It;s gonna be a lot of toying in the Console.

[zogthegreat]

Really nice build you have going here. (sigh) Must be nice to have the cash for all of those SSD's!

[Windows7ge]

1 minute ago, zogthegreat said:

Must be nice to have the cash for all of those SSD's!

Yeah, the cash...totally, it's not like I'm currently on the hunt for a new job because I'm currently broke or anything...

<.<

>.>

...

[zogthegreat]

1 minute ago, Windows7ge said:

Yeah, the cash...totally, it's not like I'm currently on the hunt for a new job because I'm currently broke or anything...

<.<

>.>

...

(chuckle) Seems to be the life story of most modders! 

[Windows7ge]

For as expensive as this project has been I did not expect 14 people to take interest.

 

I'm now working on a new much less expensive but still unique in it's own way server project. If anyone wants to follow along they're welcome to.

 

 

[Ben17]

4 hours ago, Windows7ge said:

For as expensive as this project has been I did not expect 14 people to take interest.

 

I'm now working on a new much less expensive but still unique in it's own way server project. If anyone wants to follow along they're welcome to.

 

 

 nice

Looking forward to it