Video Game Studio: NAS, Matchmaking & Build Machine

[CradleGames]

Good morning,

 

We are a small video game company working on what could be called "Dark Souls in Space".

So far, we went by with a SVN on a NAS made from normal HDD in a 12 years old dusty PC.

We were making backup by hand on a external HDD I kept at home.

 

We have been hiring, and now we are looking for something more stable, safer, faster, easier to remote control, able to do automatic backup, and the ability to schedule game builds on PC, Mac and Linux.

Considering we can only build for Mac on a Mac, we will obviously need a setup with more than 1 computer. 

 

I've been looking at a 2U server hotswappable server chassis for hosting the depot, with proper ECC memory and NAS HDDs.

However... my knowledge pretty much stop there. I have no clue what is good and what isn't.

 

The SVN/Matchmaking server doesn't need much CPU, but mostly disk space and responsiveness.

The build machine some good CPU, but not so much disk space, as its task will be to pull a changelist off the depot, open it, compile it, and push back the result. 

 

Where should I begin with all this?

[GDRRiley]

look at supermicro and maybe pic up a 2U with something like dual epyc in this chasis 2023US-TR4. This gives you enough for 8 HDD in a main raid 6 for bulk storage and then spots for 6 sata SSDs using some for SVN/matchmaking and Build spaces. 

This would give you up 64 cores and 128 threads to share across different uses. Or you could divide this all out for redundancy across a few servers.

You still should have offsite backup as raid isn't a replacement for that. 

 

[CradleGames]

Oh, totally agree that offsite backup is not a choice. It must exists. However, I'm probably looking at a way to automatize it.

 

Isn't a dual epyc horribly overkilled? Unless you mean having the depot AND the build machine on the same chassis? 

[GDRRiley]

1 hour ago, CradleGames said:

Oh, totally agree that offsite backup is not a choice. It must exists. However, I'm probably looking at a way to automatize it.

 

Isn't a dual epyc horribly overkilled? Unless you mean having the depot AND the build machine on the same chassis? 

yep i'm saying put all into 1 box and VM and pass hardware to what needs it. 

[CradleGames]

12 hours ago, GDRRiley said:

yep i'm saying put all into 1 box and VM and pass hardware to what needs it. 

How would the depot VM and the build machines VMs communicate with each other? I assume VMs cannot easily share data?

For the records, I need to build Windows, Mac, Linux, Xbox One, PS4 and Switch.

By the way... How the hell do I handle the Mac part? The last Xserver was made in 2009. The last Mac Pro was in 2012. Sure not gonna blow 7k on a iMac Pro.

[GDRRiley]

1 hour ago, CradleGames said:

How would the depot VM and the build machines VMs communicate with each other? I assume VMs cannot easily share data?

For the records, I need to build Windows, Mac, Linux, Xbox One, PS4 and Switch.

By the way... How the hell do I handle the Mac part? The last Xserver was made in 2009. The last Mac Pro was in 2012. Sure not gonna blow 7k on a iMac Pro.

for mac get a 2012 or 2010 and put in a new GPU into it and upgrade the CPU as much as you need.

VMs can talk to each other over LAN for sure. 

[CradleGames]

On 5/1/2018 at 10:29 AM, GDRRiley said:

for mac get a 2012 or 2010 and put in a new GPU into it and upgrade the CPU as much as you need.

VMs can talk to each other over LAN for sure. 

While being there... Any advice on RAID?

At first I thought going Raid 6, but some websites claims for our size of business 10 would be better.

 

Also, feeling Enteprise HDD would be overkill, and NAS would work just as well?

[GDRRiley]

42 minutes ago, CradleGames said:

While being there... Any advice on RAID?

At first I thought going Raid 6, but some websites claims for our size of business 10 would be better.

 

Also, feeling Enteprise HDD would be overkill, and NAS would work just as well?

raid 6. most nas drives are fine for this workload and deployment sizes

[AbsoluteFool]

Why not combine the matchmaking and the build machine into one server?

Let's say you used a 3Ghz+ something CPU (or two). Would be good for the build machine and probably other things aswell. Both of the machines could be is seperate VMs using Proxmox or ESXI whatever floats your goat.

 

As you say the matchmaking machine don't need alot of CPU so you can assign as much you want after need. The build machine needs more CPU, RAM.

So with a raid controller you could set-up two diffrent RAIDs. Eks. RAID 1 for matchmaking SSD drives for speed and a little redundancy. The build machine could then go on SAS drives in a completely diffrent RAID level.

 

Will give both of the machines you need in one build, instead of wasting time and money on two machines. Even with scaling this would be easier then buying two machins each time you want to scale up.

[CradleGames]

2 minutes ago, AbsoluteFool said:

Why not combine the matchmaking and the build machine into one server?

Let's say you used a 3Ghz+ something CPU (or two). Would be good for the build machine and probably other things aswell. Both of the machines could be is seperate VMs using Proxmox or ESXI whatever floats your goat.

 

As you say the matchmaking machine don't need alot of CPU so you can assign as much you want after need. The build machine needs more CPU, RAM.

So with a raid controller you could set-up two diffrent RAIDs. Eks. RAID 1 for matchmaking SSD drives for speed and a little redundancy. The build machine could then go on SAS drives in a completely diffrent RAID level.

 

Will give both of the machines you need in one build, instead of wasting time and money on two machines. Even with scaling this would be easier then buying two machins each time you want to scale up.

I need actually (excluding the Mac build machine, which I just bought a XServe 3 Dual CPU for 250$ off ebay!);

- SVN Depot

- Matchmaking

- Cloud Save Game (SQL)
- Windows Build Machine
- Linux Build Machine
- PS4 Build Machine
- Xbox One Build Machine
- Nintendo Switch Build Machine

The Depot, Matchmaking, Cloud can easily be a single computer, cheap CPU, lot of bandwidth and storage. I don't imagine the CPU ever bottlenecking anything unless I need to rebuild the RAID. Probably 2 VMs, one for the internal depot, and one for the game connectivity testing. 

Something I have to decide is how I setup the different build machines. The issue with games is that each platforms need their own local library to build, and switching from one platform to another is LONG. So probably a separated VM for each platforms. It needs a ton of RAM, not much storage, and it can easily saturate 4 cores. I don't think chaining the platforms one after the other is a good idea, as a single build can take 20-30 mins. So can a single computer handle 5 VM all building at the same time? Or should I make one for Windows/Linux, and one for the consoles?

 

So obviously I'll have to hook up something on SVN to check the current changelist every X mins, and see if a specific flag is added in the changelist description. Ex.: @Build:All, @Build:Consoles, @Build:Mac...

 

I'll also need to find a way to not saturate the depot's bandwidth while the build machines are pulling the latest changelist or commiting the built results. 

 

I know in big studio - like Ubisoft - we had like 30 build machines, and a dispatcher would assign a changelist to each of them. So every single changelist would be built. Can't afford that.

[GDRRiley]

3 hours ago, AbsoluteFool said:

snipe

wow you did read at all what I said? thanks for just rephrasing all that I said. 

Edited May 3, 2018 by GDRRiley

[AbsoluteFool]

9 hours ago, GDRRiley said:

wow you did read at all what I said? thanks for just rephrasing all that I said. 

How did i rephase you? I didn't recomend supermicro?

[GDRRiley]

1 hour ago, AbsoluteFool said:

How did i rephase you? I didn't recomend supermicro?

other than not recommending a brand you restated everything else.