supercomputer! Clustering Servers

[Scvanguard]

Hey Y'all

I'm get to play with some Servers whilst at work in my free time. i have a few servers that I am hoping to cluster together. Currently have them all hooked up to a switch and what will be the master node connecting to the outside world. Do you guys have any advice or guides on how to get this cluster started? 

 

Brandon

[manikyath]

piece of advice: if you have to ask, this is probably above your head.

 

aside from that, i'm mostly just gonna say that clustered servers are useful for less purposes than you think, only the most multithreaded of applications have any benefit.

[leadeater]

We will need much more details as to what you are doing, the term clustering is generic and doesn't provide much that we can help with.

 

• What OS or hypervisor? i.e. VMware, Linux, Windows?
• What service/application are you trying to cluster or put on this cluster?
• Are you clustering for performance or availability? Active/Active or Active/Passive

[Scvanguard]

I'm an intern for Computer Science and Computer Engineering for the University and this is something i get to do on the side. I don't have much experience in networking, as most of it is programming, testing and  which is why I think it's going to be fun! Alas, my boss thinks it's good for me to drown a little. 

 

Currently my "brother in crime" and I have CENTOS running. The plan was to get Ovirt running on top of them. I haven't started getting Ovirt running however, we have been busy. We are not going to be putting together a HA cluster, as we decided this was less fun and is what everyone at the University does. Not having much experience in Clustering I would have shot for a Beowulf cluster, but I'm open to alternatives. 

 

I'm mainly worried because conceptually it seems pretty straightforward. Yet I'm assured by my professor that this will take longer than I think. 

[tt2468]

Proxmox makes clustering easy, as it uses quorum. Go check it out! Though like @manikyath said, it might be over your head.

[RedWulf]

If you want to kick it old school I can send you the pdf my tech club used while trying to make a "beowulf" cluster to run hadoop on ubuntu. Its mostly just a simple check list so we didn't confuse anything between the few of us or skip anything. We were and still are stuck on trying to set java variables straight tho. As our mentor put it "theres some voodoo to java with linux and big data". Hadoop isn't the end all tho, theres others but thats just the one we picked. The only issues are the directories have changed abit since we started with trusty tar and the java issue, other than that its gone smoothly. 

[unijab]

What kind of cluster / to do what with them?

[Jarsky]

On 1/07/2016 at 3:45 PM, manikyath said:

aside from that, i'm mostly just gonna say that clustered servers are useful for less purposes than you think, only the most multithreaded of applications have any benefit.

for distributed computing sure, but theyre extremely useful for failovers for member patching/maintenance/issues.

[unijab]

11 hours ago, Jarsky said:

for distributed computing sure, but theyre extremely useful for failovers for member patching/maintenance/issues.

and load balancing, disaster recovery, etc

[leadeater]

11 hours ago, Jarsky said:

for distributed computing sure, but theyre extremely useful for failovers for member patching/maintenance/issues.

 

13 minutes ago, unijab said:

and load balancing, disaster recovery, etc

That's the big problem, we don't know what the desired workload is so we're all throwing darts and hoping we hit the bulls-eye. I mean is this for hosting VMs, running scientific calculations/simulations, rendering/ transcoding video and audio, distributed file storage?????? Or a combination of??

 

The mention of Ovirt at least hints at some kind of requirement for hosting VMs but then that raises more questions, why Ovrit and why not a dedicated hypervisor OS? Nutanix Community Edition would be in my view a better choice, biased as we use it at work but the paid version. And to show my even stronger bias I'm a big user of VMware and have been for years so I have very high expectations of products, no time for 'that feature is coming soon' or 'if you fudge it it will work'.

 

Then a Beowulf cluster is also mentioned which is a completely different thing all together, I'm going to guess VMs will be used as compute nodes for it due to the mention of Ovirt.