DNA Processing Server... A $30,000 adventure

[Apollod]

So I fairly recently started working in a research lab that works with microbial genomes. Basically it’s a ton of data processing. My boss and I talked about one of my hobbies being pc building, and he asked me if I would be interested in leading the search for a new server. With the rough budget of 30,000 USD, I pretty much agreed just for the adventure. Only thing is, I’m very new to the server space.

 

Here’s the key specs he’s after-

At least 80 threads (more would be better)

At least 1TB RAM (once again, more is good)

Somewhere above 150TB of storage (In a RAID array, ~200TB ideally)

 

We already run two other servers with less storage that are 24c/48t and 40c/80t Do you wonderful people have any recommendations on where to look to either get the right parts or matching servers to the specs? Dell Enterprise and HPE just aren’t cutting it, especially since I know Epyc Rome exists. I’ve thought about either doing it all in one like the existing servers or setting up two separate processing and storage servers. What do you all think?

[MartinKweh]

Just go for epyc. That will literally give you up to 128 threads and support for up to 4tb ram at a lower cost than going with xeons.

[Jarsky]

Maybe consider the Gigabyte R282 series, with Epyc 7002 series CPU's?

You could add an external disk shelf for storage via external SAS connectivity. 

 

 

[leadeater]

4 minutes ago, Apollod said:

Dell Enterprise and HPE just aren’t cutting it, especially since I know Epyc Rome exists.

They both offer EPYC Rome servers e.g.

 https://www.hpe.com/nz/en/product-catalog/servers/apollo-systems/pip.hpe-apollo-35-v2-system.1012136313.html

https://www.hpe.com/nz/en/product-catalog/servers/proliant-servers/pip.hpe-proliant-dl385-gen10-server.1010268408.html

 

Apollo is HPE's HPC product line and that Apollo 35 system supports 4 server nodes in a single chassis, 1TB per node is supported.

 

Ideally you would have a storage server or storage array but $30K isn't going to stretch too far when you need 1TB ram and two high end CPUs.

 

A Dell server with two 7532 CPUs and 1TB of ram is $27k without storage.

[Radium_Angel]

1 hour ago, Apollod said:

research lab that works with microbial genomes.

I work in a place that does the same thing.

When I get back to the office, I'll list the specs of our system(s)

[GDRRiley]

R282-Z90

dual 64 cores

2tb of ram

12 16tb seagate drives

on some server buying sites is about 35.5k

[Apollod]

12 hours ago, leadeater said:

They both offer EPYC Rome servers e.g.

 https://www.hpe.com/nz/en/product-catalog/servers/apollo-systems/pip.hpe-apollo-35-v2-system.1012136313.html

https://www.hpe.com/nz/en/product-catalog/servers/proliant-servers/pip.hpe-proliant-dl385-gen10-server.1010268408.html

 

Apollo is HPE's HPC product line and that Apollo 35 system supports 4 server nodes in a single chassis, 1TB per node is supported.

 

Ideally you would have a storage server or storage array but $30K isn't going to stretch too far when you need 1TB ram and two high end CPUs.

 

A Dell server with two 7532 CPUs and 1TB of ram is $27k without storage.

I must’ve missed the HP ones. Dell just doesn’t seem like we can get everything we want for the right budget. Yea the 30k sounds like a lot, but after two CPUs you’re already up to 10k

[Apollod]

12 hours ago, Jarsky said:

Maybe consider the Gigabyte R282 series, with Epyc 7002 series CPU's?

You could add an external disk shelf for storage via external SAS connectivity. 

 

 

Would doing disks this way basically all act as one machine? Could it be worth getting a separate NAS and processing server? (Possible expansions are down the road)

[GDRRiley]

58 minutes ago, Apollod said:

Would doing disks this way basically all act as one machine? Could it be worth getting a separate NAS and processing server? (Possible expansions are down the road)

I'd go with separate servers so the compute can just have a SSD cache on board with then a big file storage server.

but the budget will have to go up.

45 drives has large array cheap servers.

 

1 hour ago, Apollod said:

I must’ve missed the HP ones. Dell just doesn’t seem like we can get everything we want for the right budget. Yea the 30k sounds like a lot, but after two CPUs you’re already up to 10k

hp also has the Dl385 gen 10+ which are epyc 2nd gen but you'll need to contact for pricing.

[Apollod]

5 hours ago, GDRRiley said:

I'd go with separate servers so the compute can just have a SSD cache on board with then a big file storage server.

but the budget will have to go up.

45 drives has large array cheap servers.

 

hp also has the Dl385 gen 10+ which are epyc 2nd gen but you'll need to contact for pricing.

Having all the files in one place would be amazing, I may have to make some budget magic happen though! For caching, is there any typical rule for how large the cache should be? Would it be an issue if multiple users were using the server at once?

[GDRRiley]

Just now, Apollod said:

Having all the files in one place would be amazing, I may have to make some budget magic happen though! For caching, is there any typical rule for how large the cache should be? Would it be an issue if multiple users were using the server at once?

cache should be bigger than ram. it just comes down to your usage.

you'd rather have enough ram but if you are running projects back to back it could help get files off the memory so they can get written to disk.

you are going to want 40 or 100gb network.

[leadeater]

23 minutes ago, Apollod said:

Would it be an issue if multiple users were using the server at once?

You got any job queue management like Slurm? You'll probably want to look in to something like that at some point if you do go down a dedicated storage server route with compute nodes etc.

 

Also how are you accessing the files? Are the running jobs going to directly call them from the storage server or are you going to stage the required files for the job on local cache on the compute node running the job?

[Jarsky]

7 hours ago, Apollod said:

Would doing disks this way basically all act as one machine? Could it be worth getting a separate NAS and processing server? (Possible expansions are down the road)

Yes it would act as one machine, but the idea of doing a disk shelf and using SAS is that it eliminates the need for iSCSi or FCoE or something like that.

You'd need to consider your budget more carefully doing dedicated storage as you'll need to consider the networking. Ideally for the speed you need for storage access you'll be wanting SFP+ or QSFP(+) switching.

[GDRRiley]

2 minutes ago, Jarsky said:

Yes it would act as one machine, but the idea of doing a disk shelf and using SAS is that it eliminates the need for iSCSi or FCoE or something like that.

You'd need to consider your budget more carefully doing dedicated storage as you'll need to consider the networking. Ideally for the speed you need for storage access you'll be wanting SFP+ or QSFP(+) switching.

you can get older 40gb switches 2nd hand now for cheap.

this would be a good place to save some money and by 2.

[Dutch_Master]

For networking at those speeds, consider DAC if the compute and storage machines are physically close (same rack or so) or fiber-optic. On Aliexpress I found fiber-optic switches (4 or 8 SFP ports, it looks like unmanaged) but I suspect only up to 1 Gb/s, so of little use in this case. Maybe on Alibaba (=enterprise version of Aliexpress) more suitable switches can be had for cheap-ish. No idea on quality though.

[Apollod]

On 2/26/2020 at 9:23 PM, leadeater said:

You got any job queue management like Slurm? You'll probably want to look in to something like that at some point if you do go down a dedicated storage server route with compute nodes etc.

 

Also how are you accessing the files? Are the running jobs going to directly call them from the storage server or are you going to stage the required files for the job on local cache on the compute node running the job?

We've haven't made a call on how the files will be managed yet, we've got a few months before the grant money comes in. I know local caching can really improve the workflow on the compute server, but occasionally we deal with up to 10 files that are 8-14 gigabytes by themselves. I'm not sure how that would work on the cache. In that capacity I wonder what the performance impacts would be for direct calling.

 

We don't use any job queues currently, its never been something we've even considered to my knowledge. I don't know much about that area. We can typically run several workloads at once with a couple notable exceptions that will easily take up our entire capacity. We currently run Ubuntu on our two servers and we work through SSH terminal to run our jobs and such. What's the overhead and benefits of the job queue? Would it prevent us from running things simultaneously?

[leadeater]

2 hours ago, Apollod said:

What's the overhead and benefits of the job queue? Would it prevent us from running things simultaneously?

More that it'll allow you to automate the process and if you do go with local storage cache on the compute nodes copy the required files over for the job, but this is still something you have to get going it won't just work.

 

A cluster job manager will run as many jobs as you define. It's not actually something you need with only 1-3 compute nodes but when you have 3 or more it does make it simpler to manage your jobs and allocate them to servers, manually doing it will get to a point where it gets annoying then transition to unmanageable reliably.

 

I'd suggest on the side spinning up 4 VMs on a workstation or something and setup a cluster manager and 3 compute instances and give Slurm a try, don't worry about speed or letting the jobs complete or anything just do it to get working understand of Slurm and if it's even worth pursuing further for you.