insane 16 GPU server rack build concept

[CelluloidRacer2]

Foreword: I realize this is incredibly impractical and insane, but I want to see how many holes y'all can poke in this idea to see just how impossible it is.

 

PCPartPicker: https://ca.pcpartpicker.com/list/2dG7LD

Note that PCPartPicker has some misinformation about the processor, thinking it's incompatible with the mobo, googling revealed otherwise

 

The specs given to each VM are as follows:

8 GBs of RAM

4 Cores

2 TBs of storage

One nVidia GTX Titan X

 

The idea was to build "16 gamers one rack", but to do so I found some obstacles (here are the ones I've at least partially tackled):

 

1q: How do you fit 16 GPUs in a chassis?

1a: Short answer is to seperate them see this to understand what exactly I mean

 

2q: How do you connect 16 GPUs? There's only so many slots!

2a: Backplanes! (Although it does mean chopping bandwidth, which I'm still trying to figure out)

 

3q: How do you get so many processors/threads for vms?

3a: Go dual sockets with xeon processors, lots of processors/threads!

 

4q: How do you get each user USB connections?

4a: Use a hub or dock

 

Some issues I haven't dealt with:

Graphics card bandwidth

USB/IO Bandwidth

Hypervisors may not be able to see the GPUs and thus may not be able to assign them to VMs

Storage interfaces (enough IO and ports?)

 

 

Edit was to make the title more accurate

[AndersT2]

First question is WHY?

 

And it almost sounds cheaper and easier to build 16 separate machines With 4 cores and a 1080 Ti which gives almost just the same gaming Power as the Titan, but for a lot less Money.

[CelluloidRacer2]

Just now, AndersT2 said:

First question is WHY?

 

And it almost sounds cheaper and easier to build 16 separate machines With 4 cores and a 1080 Ti which gives almost just the same gaming Power as the Titan, but for a lot less Money.

Just because! No particular reason, and you are right, the overall cost is significantly higher than building 16 individual rigs

[samcool55]

The more reasonable option would be 2 racks with 8 gpu's each.

I also think you are underestimating the scale of doing this...

 

You will need to call an electrician and add wiring to your house to make sure those things get enough power.

I mean you are basically entering the enterprise world with such setup and looking at a 5000W+ setup under load. (probably 6000-7000W)

Cooling is another thing, an AC is a requirement at that point or you won't be able to keep the room cool enough, and they also eat up a lot of power.

 

If you really want this, send an e-mail to supermicro, tell them you need racks with the specs you mentioned and they will probably send you an invoice back.

[CelluloidRacer2]

Just now, samcool55 said:

The more reasonable option would be 2 racks with 8 gpu's each.

I also think you are underestimating the scale of doing this...

 

You will need to call an electrician and add wiring to your house to make sure those things get enough power.

I mean you are basically entering the enterprise world with such setup and looking at a 5000W+ setup under load. (probably 6000-7000W)

Cooling is another thing, an AC is a requirement at that point or you won't be able to keep the room cool enough, and they also eat up a lot of power.

 

If you really want this, send an e-mail to supermicro, tell them you need racks with the specs you mentioned and they will probably send you an invoice back.

Good to know!

I'm not seriously considering doing this, it's more a "what would it take" kind of thing