K4000 SLI mod?

[Flufflebutt]

Hello guys,

I've been searching through the internet for a few days now but didn't quite find what I'm looking for.

What I'm trying to do is run around 50VMs on a server which has 2x Quadro K4000 installed.

Right now I'm at around 26 VMs with 100% VRAM usage so I got a second card to "split the load". In the end I don't really give a damn if the cards run in SLI or not, I just need 128MB VRAM for each VM.

Even though I've heard "K4000 doesn't support SLIiiiii" more often than I can count now, I've heard that it's more of a software lock.

That thought finally brings me to you "LTT guys", since I'd point anyone in need of modded (unsigned) drivers right here.

 

Please spare me the obligatory "a single K5000 is better than 2x K4000" and the like. I asked how to do SLI with 2x K4000 and not what would work better or would be easier.

If you can help me, PLEASE DO. But if you can't save mine and your time please.

(It HAS TO BE 2x K4000 since I could get those dirt cheap and because my Server only fits single slot GPUs. It's a Proliant DL380 G7)

[WhitetailAni]

If you want to run SLI to use more VRAM, know that SLI does not double your VRAM. It mirrors it, so the VRAM on card 1 will be (mostly) the same as the VRAM on card 2.

If you want to dedicate 128MB of VRAM to each VM, with 3GB of VRAM, you can run 24, and with 6GB (running your two Quadro K4000s separately) you can run 48.
Can you not just run the cards separately?

[Flufflebutt]

I would LOVE TO just run them seperately but unfortunately VMWare Workstation Pro doesn't let me.

The statement of Citrix support is more or less "buy VSphere enterprise pro" which costs 4.5k...

[emosun]

Also your server board might not support sli. 

 

But yeah sli only clones the vram so you would need to be running them separately to use all their vram for unique data between the two.

[Electronics Wizardy]

The program needs sli support to use it, and I'm pretty sure SLI can't be used here, or won't help you here. 

 

What are the vms doing, id just let them use the virtual gpu on the cpu and don't use the gpu here. I have systems running 50+ vms with no gpu at all.

 

What are the full system specs?

[Flufflebutt]

well makes sense since the bandwith wouldn't even be CLOSE to what RAM sharing would need... I think this new stuff supports that one? NvLink?

I was actually wondering about the board not supporting SLI since the DL380 sure as shit isn't made for such shenanigans..

My original plan actually was to just "give" each GPU 20-25VMs depending on the load but thought that might not be doable in VMWare.

To be frank, the VMs are farming game accounts I'm (rather unsuccessfully) selling, so the load is pretty much constant

 

full specs:

2x Xeon X5650 (going to 2x X5690 this week)

8x 16GB DDR3 Samsung PC3L-8500R DIMM 4Rx4

2x Quadro K4000

[Electronics Wizardy]

4 minutes ago, Flufflebutt said:

well makes sense since the bandwith wouldn't even be CLOSE to what RAM sharing would need... I think this new stuff supports that one? NvLink?

I was actually wondering about the board not supporting SLI since the DL380 sure as shit isn't made for such shenanigans..

My original plan actually was to just "give" each GPU 20-25VMs depending on the load but thought that might not be doable in VMWare.

To be frank, the VMs are farming game accounts I'm (rather unsuccessfully) selling, so the load is pretty much constant

 

full specs:

2x Xeon X5650 (going to 2x X5690 this week)

8x 16GB DDR3 Samsung PC3L-8500R DIMM 4Rx4

2x Quadro K4000

Yea SLI won't work here then, its made for games and 3d apps only, not this use.

 

Really buying a faster gpu is your best option here probably.

 

Otherwise, can you use the software gpu, should be fine here. Or try anouther hypervisor. Hyper-v should be able to do this for free(other than guest os licensing). MIght need 2016 as they have taken out the virtual gpu in favor of gpu partitioning due to security issues.

 

You could also run anouther hypervisor on the host like proxmox, then make 2 vms with exsi on it, and run half the vms on each and pass a gpu to each virtual hypervisor.

[Flufflebutt]

Right now I'm actually just running Win10 on the thing but HyperV was actually what I tried first.

Didn't manage to set it up tho or I read somewhere it doesn't work for some arbitrary reason, I don't remember actually...

I've read about this "software GPU" thing before as well, but whenever I ask google, I just find vGPU, vSphere or Nvidia GRID cards.

So I'm more or less fighting on lost ground I suppose? 

9 minutes ago, Electronics Wizardy said:

You could also run anouther hypervisor on the host like proxmox, then make 2 vms with exsi on it, and run half the vms on each and pass a gpu to each virtual hypervisor.

VMception... I like your way of thinking... this might actually work since as far as I know there is no issue at all with dedicating two GPUs to two VMs.

 

 

edit:

On the long run I plan to go for a Radeon Pro WX7100 anyway so a temporary solution like that would be just fine.

[Electronics Wizardy]

2 minutes ago, Flufflebutt said:

Right now I'm actually just running Win10 on the thing but HyperV was actually what I tried first.

Didn't manage to set it up tho or I read somewhere it doesn't work for some arbitrary reason, I don't remember actually...

I've read about this "software GPU" thing before as well, but whenever I ask google, I just find vGPU, vSphere or Nvidia GRID cards.

So I'm more or less fighting on lost ground I suppose? 

You should be able to do this with no gpus in the system and use cpu power for all of it. Have you tried that.

 

3 minutes ago, Flufflebutt said:

edit:

On the long run I plan to go for a Radeon Pro WX7100 anyway so a temporary solution like that would be just fine.

Why not get a consumer grade card? Your not using the pro card features here anyways.

[Flufflebutt]

Just now, Electronics Wizardy said:

You should be able to do this with no gpus in the system and use cpu power for all of it. Have you tried that.

My CPU is already "idling" at 80-90% with 35VMs so there isn't much left for graphics I presume... I'll have to see how the 5690s will perform because that might just work as well. 

 

2 minutes ago, Electronics Wizardy said:

Why not get a consumer grade card? Your not using the pro card features here anyways.

I'll do that in a heartbeat but just didn't manage to find a single slot GPU with enough VRAM which doesn't cost as much as the entire rest of the server... Bad times to go GPU hunting... I'm happy that I got those K4000s for 55 and 70€. At least I won't lose much when I have to sell those again.

 

 

Well, I got at least a few things I have to try out now, thanks a forking lot for that already.

[Electronics Wizardy]

5 minutes ago, Flufflebutt said:

y CPU is already "idling" at 80-90% with 35VMs so there isn't much left for graphics I presume... I'll have to see how the 5690s will perform because that might just work as well. 

THen Id just get a second system then. Might be best to have multiple smaller systems instead of one huge one here.

 

If you care about power, id get something newer, those x5690s are power hungry ones.

[Flufflebutt]

Just to close this off...

 

It worked just fine with one GPU and 40 VMs. It took the "missing" VRAM just from system memory without issue.

 

Regarding power consumption, the whole system draws around 620W at 100% and idle around 315W. 

I'd love to use a newer system but those still cost quite a lot and I got this one pretty damn cheap. (Around 500€ with all upgrades I made thus far.)

 

I'll probably get a Radeon Pro WX7100 soon, then I'm golden for the time being =)