Raspberry Pi Cluster - How To Build?

[WillWam]

Here's what I have

 

• 3 SD cards
• 3 Raspberry Pi B+s
• All the necessary cables, power, HDMI, etc.
• 3 Instances of Raspian Stretch Desktop

 

How can I turn this into a Raspberry Pi cluster? They're all setup on the same wireless.

 

Defining "Pi Cluster"

 

To be clear, what I mean by "pi cluster" is all 3 of my pis connected wirelessly sharing RAM, CPU (if possible), and SD card storage. Asking for a friend :)

[Creed1]

I was actually doing some research myself on this topic earlier today. What I ran into was a project called OctaPi. You can do things like develop a Python3 applications and deploy them to the cluster. 

[WillWam]

1 minute ago, Creed1 said:

I was actually doing some research myself on this topic earlier today. What I ran into was a project called OctaPi. You can do things like develop a python3 applications and deploy them to the cluster. 

Yeah, but how do you connect the Pis?

[Creed1]

Just now, WillWam said:

Yeah, but how do you connect the Pis?

Well, first you will need one of those raspberry pi's to be setup as the client for the cluster. Then the rest of those raspberry pi's will be used as servers. The client and server setup should be straightforward. All you need to do is install all the necessary dependencies and download the necessary Octapi files for the server and client. 

[WillWam]

9 minutes ago, Creed1 said:

Well, first you will need one of those raspberry pi's to be setup as the client for the cluster. Then the rest of those raspberry pi's will be used as servers. The client and server setup should be straightforward. All you need to do is install all the necessary dependencies and download the necessary Octapi files for the server and client. 

Ok, I'll look at it. Will this include merging RAM, CPU, and SD storage?

[WillWam]

13 minutes ago, Creed1 said:

Well, first you will need one of those raspberry pi's to be setup as the client for the cluster. Then the rest of those raspberry pi's will be used as servers. The client and server setup should be straightforward. All you need to do is install all the necessary dependencies and download the necessary Octapi files for the server and client. 

Also, is there a way to do this without an Ethernet cable? I'd like to do it wirelessly.

[colonel_mortis]

Most compute clusters, including I believe the OctaPi mentioned above, don't provide an abstraction of being the same computer. To run multiple CPUs as if they are physically connected, and to allow them to access each other's memory, you generally need them to be on the same motherboard. In particular, the latency for memory accesses needs to be exceptionally low. There are datacentres that do stuff like that, but they have a lot of very expensive hardware and specialised software to achieve it.

 

What most clusters (including the OctaPi) actually do is let you distribute a task across multiple compute nodes, which run independently, completing the tasks that they are assigned and returning the results to a central location. This works really well, provided that the tasks are able to be split up like that (and there are some standard algorithms like Map-Reduce that are designed for such architectures).

 

I wouldn't know where to start if you are set on setting the pis up to appear as a single compute unit with one OS (nor do I think that would perform at all well). The OctaPi guide does look like a god way to set up a compute cluster though, just not quite the one you had in mind.

[WillWam]

1 minute ago, colonel_mortis said:

Most compute clusters, including I believe the OctaPi mentioned above, don't provide an abstraction of being the same computer. To run multiple CPUs as if they are physically connected, and to allow them to access each other's memory, you generally need them to be on the same motherboard. In particular, the latency for memory accesses needs to be exceptionally low. There are datacentres that do stuff like that, but they have a lot of very expensive hardware and specialised software to achieve it.

 

What most clusters (including the OctaPi) actually do is let you distribute a task across multiple compute nodes, which run independently, completing the tasks that they are assigned and returning the results to a central location. This works really well, provided that the tasks are able to be split up like that (and there are some standard algorithms like Map-Reduce that are designed for such architectures).

 

I wouldn't know where to start if you are set on setting the pis up to appear as a single compute unit with one OS (nor do I think that would perform at all well). The OctaPi guide does look like a god way to set up a compute cluster though, just not quite the one you had in mind.

It's not about performance, I'm just experimenting.

[Cheezdoodlez]

13 minutes ago, WillWam said:

Also, is there a way to do this without an Ethernet cable? I'd like to do it wirelessly.

If you look at the octapi guide it only talks about wifi and not ethernet, so yes.

[WillWam]

Just now, Cheezdoodlez said:

If you look at the octapi guide it only talks about wifi and not ethernet, so yes.

Aight. Still reading up on it.

[tbake0155]

16 hours ago, WillWam said:

It's not about performance, I'm just experimenting.

colonel_mortis is  essentially correct.  If you wanted to have all of the resources appear as one computer, you would have to run some type of simulation layer on either a 'master' machine or one of the workers in the cluster that would then behave as a master and a worker (probably not ideal).  This would be an OS-like piece of software, and would most likely severely hinder performance, which is why you probably won't find something like that available, but in theory you could write something up yourself (it would be a huge undertaking).

 

In general, leveraging a cluster requires you to write multi-threaded applications that can use the resources of several machines, usually by working individually and then combining the results when all of the workers are done.  This means you need an algorithm that can be tackled with multiple threads simultaneously without the threads depending on each other (or at least minimally so, as they could use shared memory and access the same variable, but not at the exact same time due to locking), which is pretty normal in scientific computation but not necessarily for all cases.  There is cluster software available like Slurm that you can install to manage your cluster more easily.  Whatever you do, you will need to use a library like MPI when writing your code so that your nodes can work together.

 

Take a look at this: https://shop.pimoroni.com/products/cluster-hat

 

You might go through the instructions for setting that up to get an idea of how to proceed.  You could probably even use the modified raspian image provided by the manufacturer, but you would have to make some changes since the instructions are for setting up USB network connections rather than ethernet.