Supercompute chat

[GDRRiley]

I get to play with supercomputers at my internship and I'm sure people want to talk more about these insanely large clusters of machines. ( @leadeater)

Be nice and realize that for those who work at facilities they maybe under NDA so we can't talk about everything

[leadeater]

1 minute ago, GDRRiley said:

Be nice and realize that for those who work at facilities they maybe under NDA so we can't talk about everything

No, I demand you tell me EVERYTHING!! 

 

There is one thing I want to actually know but I can find that out myself, or feel it out with HPE, and that's when EPYC3 is actually going to be available. We've got a bunch of hardware to replace and I really want to get quotes for Zen 3 systems.

[GDRRiley]

I really want my work to create a much better video than this when our new system Perlmutter comes in.

24 water cooled cabnets (in between not each server) for compute and then more for the 35PB 5TB/s flash arrray

12 CPU only with 2 Milan and 512gb of memory

12 GPU+CPU with 1 Milan 256gb of memory and 4 A100

which gives us 6000 GPUs and 7500 Milan CPUs each with 256gb of ram

Public dates are late 2020 for GPU cabinets+ flash and mid 2021 for CPU

 

our older system CORI is the largest public deployment of Xeon Phi

with 10k Xeon Phi Processor 7250 each with 96gb of ram + (16gb of onboard MCDRAM)

and 2.4k Xeon Processor E5-2698 v3 each with 128gb of ram

20 large memory nodes with dual EPYC 7302 and 2tb of ram

18 GPU test nodes with 2 Xeon Gold 6148 Processor with 384 GB and 8 V100

 

Cori has 28PB of scratch disk and 1.8PB of scratch flash at 1.5TB/s

we've got 65PB of disk storage and 200PB+ of tape

[GDRRiley]

8 minutes ago, leadeater said:

There is one thing I want to actually know but I can find that out myself, or feel it out with HPE, and that's when EPYC3 is actually going to be available. We've got a bunch of hardware to replace and I really want to get quotes for Zen 3 systems.

Server the home has had it in the labs for a while. I think he said back in OCT or NOV that Hyperscales and some other clients were getting it

I can't say anything more

 

[leadeater]

2 minutes ago, GDRRiley said:

Server the home has had it in the labs for a while. I think he said back in OCT or NOV that Hyperscales and some other clients were getting it

I can't say anything more

Well I already know they are out there, some of who have them, who likely has them. But I want to know when I can try and get them. Problem is I may still not be able to swing it as for us introducing completely different hardware architecture in to our environment is a bit of a pain, VMware/ESXi yada yada.

 

We do have approval to create a new HPC cluster this year so it'll be a lot easier for that, but it's only a very small purchase. Not a lot of money to throw around this year but if the uptake of a centrally managed ITS cluster is good expanding it down the line will get funding. Right now the biggest HPC clusters are managed and maintained by the departments like the Institute of Life Sciences (or w/e they are called now, it's not that anymore). There's 3 totally separate large clusters like this, all massively over demanded so many go without and get pushed towards NeSI but there is very little support within the university to teach and guide post grade students and researchers on how to interact with a proper workload managed cluster, NeSI use SLURM. Basically there is a ton of inequality that goes on, those who have and those who don't have,

[GDRRiley]

21 minutes ago, leadeater said:

Well I already know they are out there, some of who have them, who likely has them. But I want to know when I can try and get them. Problem is I may still not be able to swing it as for us introducing completely different hardware architecture in to our environment is a bit of a pain, VMware/ESXi yada yada.

 

We do have approval to create a new HPC cluster this year so it'll be a lot easier for that, but it's only a very small purchase. Not a lot of money to throw around this year but if the uptake of a centrally managed ITS cluster is good expanding it down the line will get funding. Right now the biggest HPC clusters are managed and maintained by the departments like the Institute of Life Sciences (or w/e they are called now, it's not that anymore). There's 3 totally separate large clusters like this, all massively over demanded so many go without and get pushed towards NeSI but there is very little support within the university to teach and guide post grade students and researchers on how to interact with a proper workload managed cluster, NeSI use SLURM. Basically there is a ton of inequality that goes on, those who have and those who don't have,

I have no idea other than to guess summer 2021

 

oh fun. I assume 100gb/200GB IB (because unless its HPE/cray your not getting slingshot), then its down to node type, assuming RU isn't a constraint and when you order a cluster P sku maybe cheaper

IDK if your work likes blades otherwise DL325 G10 plus/ Dl385 G10 plus are you HPE options

supermico blades for H12 could work but that gives you ether 1 CPU or 1 CPU+1 2 slot or 2 1 slot GPUs

[leadeater]

7 minutes ago, GDRRiley said:

I have no idea other than to guess summer 2021

Man I want to slap you and every American so hard when you do this lol. Our ancestors came up with a calendar and you're all like "nah bro, lets use something that is totally opposite for half the planet" 

 

What's Summer 2021? Like June to Sep ish?

 

7 minutes ago, GDRRiley said:

I assume 100gb/200GB IB (because unless its HPE/cray your not getting slingshot)

Don't know yet, most likely RDMA Ethernet though not IB. Will be HPE Apollo, well also likely. Luckily not under RFP for it yet but still have to show that it's vendor fair and neutral, which it will be.

[GDRRiley]

Just now, leadeater said:

Man I want to slap you and every American so hard when you do this lol. Our ancestors came up with a calendar and you're all like "nah bro, lets use something that is totally opposite for half the planet" 

 

What's Summer 2021? Like June to Sep ish?

 

Don't know yet, most likely RDMA Ethernet though not IB. Will be HPE Apollo, well also likely. Luckily not under RFP for it yet but still have to show that it's vendor fair and neutral, which is will be.

deal with it

I'd say may-AUG so Q3

 

if there isn't much of a cost difference maybe look at slingshot if its HPE apollo, I wish more venders would bid for stuff like this but no one does. Like I know we are paying IBM way to much for our disk storage system.

[leadeater]

2 minutes ago, GDRRiley said:

if there isn't much of a cost difference maybe look at slingshot if its HPE apollo

Is that even an option within the Apollo product stack? I thought it was only for Cray.

 

We kind of have to be careful not to over engineer/over shoot this one as it's scope is to be a middle service offering below that of NeSI, cost and commodity optimized. We don't want to create a competing service to NeSI as for one we couldn't anyway and we already bulk fund thousands/hundreds of thousands of compute hours from them so if someone needs that it's there to be used.

 

The departmental clusters hardware wise are all very old and funding to replace them outright just isn't there, one is getting expanded this year but I have no idea what they are getting yet. It's these that we are targeting to replace with the central service on a collectively larger and more fair access platform. And when I say it's going to be small I really do mean small, single rack small, with other services like bulk storage not within that.

 

14 minutes ago, GDRRiley said:

I wish more venders would bid for stuff like this but no one does. Like I know we are paying IBM way to much for our disk storage system.

IBM Spectrum Scale? But yea it's IBM, it'll be $$. You should have Dell knocking on your door trying to sell Isilon at least, we get that a lot. Not sure how Netapp would fit in but I know they get used at least a little bit, we have Netapp but just for core business stuff.

 

I thought you also had some DDN stuff or am I miss remembering? Never actually been hand on with any of their stuff.

[GDRRiley]

21 minutes ago, leadeater said:

Is that even an option within the Apollo product stack? I thought it was only for Cray.

IDK HPE site is a mess of marketing wank

Quote

We kind of have to be careful not to over engineer/over shoot this one as it's scope is to be a middle service offering below that of NeSI, cost and commodity optimized. We don't want to create a competing service to NeSI as for one we couldn't anyway and we already bulk fund thousands/hundreds of thousands of compute hours from them so if someone needs that it's there to be used.

 

The departmental clusters hardware wise are all very old and funding to replace them outright just isn't there, one is getting expanded this year but I have no idea what they are getting yet. It's these that we are targeting to replace with the central service on a collectively larger and more fair access platform. And when I say it's going to be small I really do mean small, single rack small, with other services like bulk storage not within that.

so then that makes me think supermicro/ gigabyte then as asrockrack is a little new. but those would have to come from local SI I think

so maybe 40 nodes if its 1U or 80 if you go blade, thats 1 racks worth.

 

I'm assuming budget under 2 million for the entire cluster?
that could get 2 racks with supermicro H12 blades (64core P epyc and 256gb of ram)  and some 100gb switches but thats without a real support plan

Spoiler

SuperBlade SBE-820J

20 7702P 256gb (8x32) 0 local disk

2 25gb internal switch with 4x100gb out

~175k from thinkmate per server so

1.75 million

10 of those in 2 racks means

80 100gb ports likely means you want 2 switches per rack if they are 100gb

~35K Aruba 8325-32C x4

~140k

 

and yes I'm crazy enough to plan this out

 

Quote

IBM Spectrum Scale? But yea it's IBM, it'll be $$. You should have Dell knocking on your door trying to sell Isilon at least, we get that a lot. Not sure how Netapp would fit in but I know they get used at least a little bit, we have Netapp but just for core business stuff.

 

I thought you also had some DDN stuff or am I miss remembering? Never actually been hand on with any of their stuff.

Yep, we had some dell stuff earlier in the history but over the last 10 years none other than a few servers.

 

We did have some DDN stuff but I think that was all for our 2015 supercomputer Edison

Edited March 3, 2021 by GDRRiley
added my crazy config

[leadeater]

15 minutes ago, GDRRiley said:

so then that makes me think supermicro/ gigabyte then as asrockrack is a little new

Gigabyte or similar vendor would only get considered if they put a proposal in, otherwise it'll be HPE (Apollo not Proliant), Dell, Lenovo and Supermicro but no 'specialty' hardware within their product ranges, just standard servers that fit within what we already do. For example there are Apollo servers we would not consider. We already have HPE Apollo 4200 LFF servers for the Ceph cluster.

 

15 minutes ago, GDRRiley said:

I'm assuming budget under 2 million for the entire cluster?

Under 1 . We will be adding on to it a lot of existing ex-production servers though. That is just for the server hardware though, everything else like networking and storage either already exist or come out of standard budgets.

 

It also means it will be losing all my servers I have been using for FAH/BOINC too, was looking forward to another run at that, think I've got triple the amount of servers than last time.

[GDRRiley]

Just now, leadeater said:

Under 1 . We will be adding on to it a lot of existing ex-production servers though. That is just for the server hardware though, everything else like networking and storage either already exist or come out of standard budgets.

oh dam that really is 1 rack.

wow my config I made was 2x the budget

[leadeater]

Just now, GDRRiley said:

oh dam that really is 1 rack.

wow my config I made was 2x the budget

Yep, minimum viable product to prove a need then expand once proven.

[GDRRiley]

Just now, leadeater said:

Yep, minimum viable product to prove a need then expand once proven.

supermicro blades are so cheap but I don't know apollo pricing, unless AMD is giving HPE CPUs it can't be much lower

[leadeater]

2 minutes ago, GDRRiley said:

oh dam that really is 1 rack.

wow my config I made was 2x the budget

Also for context the biggest cluster out in the departments, that I know the config of, is only about 1000-1500 CPUs. There is one that might be larger but the person who control that literally hates ITS with all his soul so I genuinely have no idea what they truly have. Everything I know about them HPE has told me when they come in to discuss other things, yea that's a little broken when your vendor is telling you what your own organization is doing.

[GDRRiley]

Just now, leadeater said:

Also for context the biggest cluster out in the departments, that I know the config of, is only about 1000-1500 CPUs. There is one that might be larger but the person who control that literally hates ITS with all his soul so I genuinely have no idea what they truly have. Everything I know about them HPE has told me when they come in to discuss other things, yea that's a little broken when your vendor is telling you what your own organization is doing.

CPUs or CPU cores?

1-1.5k CPUs isn't bad, looking at trends most of our systems are 10-15k CPUs

that sounds like a lovely place to work

[leadeater]

15 minutes ago, GDRRiley said:

CPUs or CPU cores?

1-1.5k CPUs isn't bad, looking at trends most of our systems are 10-15k CPUs

CPUs, but E5v1 generation. The job queue on it is ETA well over a year and it's net increasing, so that's not good.

 

16 minutes ago, GDRRiley said:

that sounds like a lovely place to work

We just leave him alone, he ignores us and everything is great. He's even in a different city  

[GDRRiley]

3 minutes ago, leadeater said:

CPUs, but E5v1 generation. The job queue on it is ETA well over a year and it's net increasing, so that's not good.

 

We just leave him alone, he ignores us and everything is great. He's even in a different city  

that sounds like a system ready for EPYC to replace it

 

lol

[GDRRiley]

12 hours ago, leadeater said:

CPUs, but E5v1 generation. The job queue on it is ETA well over a year and it's net increasing, so that's not good.

2011 isn't a bad gen. 4-12 core

Edited March 3, 2021 by GDRRiley

[GDRRiley]

someone at my work finally realized that this page is almost 10 months old with 0 update and people started asking if it will be.https://www.nersc.gov/systems/perlmutter/
So I may get to talk about more soon

[GDRRiley]

@leadeaterhttps://www.nersc.gov/systems/perlmutter/
 

my work has spilled some more beans publicly.

 

the weirdest line to me is " Phase 1's Rome CPUs will be upgraded to Milan CPUs."
so HPE is going to remove installed blades and ether swap them or swap the CPUs ~6 months after they are set to go in. okay then HPE

Edited March 8, 2021 by GDRRiley

[BuckGup]

On 3/3/2021 at 1:59 AM, leadeater said:

No, I demand you tell me EVERYTHING!! 

 

There is one thing I want to actually know but I can find that out myself, or feel it out with HPE, and that's when EPYC3 is actually going to be available. We've got a bunch of hardware to replace and I really want to get quotes for Zen 3 systems.

I may or may not work at HPE in the benchmarking lab that has this knowledge you want. 

[GDRRiley]

1 minute ago, BuckGup said:

I may or may not work at HPE in the benchmarking lab that has this knowledge you want. 

lol. tell HPE to stop taking so dam long with Milan and to just ignore any issues

[BuckGup]

18 minutes ago, GDRRiley said:

so HPE is going to remove installed blades and ether swap them or swap the CPUs ~6 months after they are set to go in. okay then HPE

Yup they like to do that. I had to swap almost 400 CPUs and then 2 days later I had to swap 28 CPUs again since they wanted to do more testing on the old CPUs. HPE is a tad messy currently as I encounter SGI, Cray, Rackable, Lenovo, HP, and now HPE hardware on the daily as a result of all the mergers and buying of other companies

[BuckGup]

1 minute ago, GDRRiley said:

lol. tell HPE to stop taking so dam long with Milan and to just ignore any issues

Ha yeah right. 

 

On 3/3/2021 at 3:07 AM, leadeater said:

Is that even an option within the Apollo product stack? I thought it was only for Cray.

Slingshot is going to be an available option on all future platforms to come from HPE/Cray