Cluster computer Arch Linux

[Eigenvektor]

5 hours ago, archiso said:

Ok, @Franck said the same thing about the main code of this project, that a different language like C#, C++, or Java would be better for this. Maybe JavaScript would work? That would be easier to put into the web front end I'm making.

I'd second that suggestion. There's a ton of libraries for these languages that should help with development of server/client applications, which is essentially what you need for this. If you've never worked with a language derived from C the learning curve might be somewhat steep though.

 

I haven't used JS for server stuff yet (mostly Angular/TS for front end), so I can't say how well suited it is for this use case. But if you're already familiar with it, you might find it easier to get going than C#, C++ or Java.

 

I could find some distributed computing libraries for JS on Google, but most of these expect the client to run in a browser, which (imho) isn't really ideal for a cluster environment, where I'd mostly expect to work with headless servers running some service(s). But maybe there is some stuff out there that is intended for servers.

[archiso]

18 minutes ago, Eigenvektor said:

I'd second that suggestion. There's a ton of libraries for these languages that should help with development of server/client applications, which is essentially what you need for this. If you've never worked with a language derived from C the learning curve might be somewhat steep though.

 

I haven't used JS for server stuff yet (mostly Angular/TS for front end), so I can't say how well suited it is for this use case. But if you're already familiar with it, you might find it easier to get going than C#, C++ or Java.

 

I could find some distributed computing libraries for JS on Google, but most of these expect the client to run in a browser, which (imho) isn't really ideal for a cluster environment, where I'd mostly expect to work with headless servers running some service(s). But maybe there is some stuff out there that is intended for servers.

Ok. @igormpsaid that what I want to do is possible in python and that I should use that. I guess I’ll try what he said and if I can’t get it to work I’ll use c# or JavaScript. Which one would you recommend I use from c# or JavaScript(I have used both), I probably have more experience with c# but that was for unity so it’s probably different.

[Eigenvektor]

5 minutes ago, archiso said:

Ok. @igormpsaid that what I want to do is possible in python and that I should use that. I guess I’ll try what he said and if I can’t get it to work I’ll use c# or JavaScript. Which one would you recommend I use from c# or JavaScript(I have used both), I probably have more experience with c# but that was for unity so it’s probably different.

I'd go for C# but that's only because I have (far) more experience with it (realistically, I'd go with Java, since that's what I've been using at work for the past few years, so I'd probably get going the fastest).

 

But ultimately either language should work, so the one you're more familiar with is probably an advantage here.

[archiso]

40 minutes ago, Eigenvektor said:

I'd go for C# but that's only because I have (far) more experience with it (realistically, I'd go with Java, since that's what I've been using at work for the past few years, so I'd probably get going the fastest).

 

But ultimately either language should work, so the one you're more familiar with is probably an advantage here.

Ok, does unity count as experience with c#?

[igormp]

54 minutes ago, archiso said:

Ok. @igormpsaid that what I want to do is possible in python and that I should use that. I guess I’ll try what he said and if I can’t get it to work I’ll use c# or JavaScript. Which one would you recommend I use from c# or JavaScript(I have used both), I probably have more experience with c# but that was for unity so it’s probably different.

You can make it work with any language. I do so daily in my work with Python:

Yesterday I managed to get around half a terabyte of data from an API without barely maxing out a single core, with the limitation being the provider's servers.

 

Before you get into any kind of clustering work, first get your stuff to not be blocking on network or disks as it is at the moment and get a single core running up to 100%. Then you can try to scale across cores, and once you have all the cores on a machine hitting 100%, you can go for multi system setups.

 

If you don't do so, the performance gains will be close to none since your CPU will be idle anyways waiting to be able to do stuff and your code will be way more complicated than needed.

[archiso]

10 minutes ago, igormp said:

You can make it work with any language. I do so daily in my work with Python:

Yesterday I managed to get around half a terabyte of data from an API without barely maxing out a single core, with the limitation being the provider's servers.

 

Before you get into any kind of clustering work, first get your stuff to not be blocking on network or disks as it is at the moment and get a single core running up to 100%. Then you can try to scale across cores, and once you have all the cores on a machine hitting 100%, you can go for multi system setups.

 

If you don't do so, the performance gains will be close to none since your CPU will be idle anyways waiting to be able to do stuff and your code will be way more complicated than needed.

Python runs on a single thread and will rarely be able to max it out. I think?

[igormp]

16 minutes ago, archiso said:

Python runs on a single thread and will rarely be able to max it out. I think?

You can use multi processing to use more than one core, and there are frameworks with python wrappers that allow you to do such work seamlessly. In my case, I mostly use Spark.

 

And you can easily max out a single core with python.

[wasab]

2 hours ago, Eigenvektor said:

 

I could find some distributed computing libraries for JS on Google, but most of these expect the client to run in a browser, which (imho) isn't really ideal for a cluster environment

There are people who run websites which will use proccessing power of client computers to farm bitcoins. The whole idea has potentials for abuse. 

 

[Eigenvektor]

6 hours ago, wasab said:

There are people who run websites which will use proccessing power of client computers to farm bitcoins. The whole idea has potentials for abuse.

Yeah, I got as much from reading the description for those libraries. "Volunteers" visiting the website contribute processing power for some distributed task. I was more interested in whether there's already a JS library that could provide support for distributed computing in a cluster scenario. One of these existing libraries might provide a starting point.