Threadripper vs multiple ARM-based Mac Minis

[Baradh]

Hey everyone,

 

I was wondering has anyone tried to connect multiple ARM-based Mac Minis to complete a collective task (for example, a detailed 3d render in Blender) like in the old days. I don't have a lot of Mac experience but I heard (and saw a couple of photos) that you could do that with the old Minis.

 

And what does it have to do with Threadripper? Let me explain...

 

In my country (Turkey, currently like the bottom of the barrel in terms of countries of the world) a complete Threadripper 3970x system is roughly the same price as 4 semi-upgraded Mac Minis (16Gb RAM, 10 gigabit Ethernet) + a semi upgraded Macbook Air (again, 16GB RAM and 512GB drive). So I was doing a thinking exercise: Has anyone compared the performance of multiple Mac Minis to the Threadripper 3970x (or 3990x)? What are the benefits? Like power consumption, multiple ready to use devices to use whatever you need? Or taking the laptop and going to another city and overseeing the render (or any other work) that is happening on the Minis? Using a remote desktop system to make changes from afar and hit "render" so while you are doing your presentation things happen in your house (or in your office)? Adding the power of the laptop to the mix when you are at place, but in other times the system works well enough? How about future upgrades, adding new Minis (M1 or newer) to the mix and improving the farm altogether?

 

Or what are the limitations? Obviously the raw power will be less than a Threadripper. But by how far? Will it compensate the power consumption, heating, maintenance of the PC case? Also the limitations of macos on ARM and unavailability of Windows (therefore lots of applications) on the system?

Or is it not feasable? Maybe ARM-based Mac Minis simply don't work together?

I don't have enough knowledge, nor the money to test these ideas out. But if anyone has an opportunity, I think this would be a nice experiment.

[Kilrah]

It would be entirely dependent on the program you're using...

[RockSolid1106]

How would you connect these Macs to each other anyway? It's kinda difficult to connect multiple machines to form a cluster. It's not as easy as connecting cables between the four and calling it a day..

[SupaKomputa]

4 m1 mac will have 32 cores and 64gb ram.

1 mini have a score of 7401(geekbench) x 4, will collectively score 29.604 (discounting any penalty or added latency).

Typical power comsuption of mac mini is around 10-20w so 80w total.

3970x scoring 22419 geekscore, but it has memory capacity up to 256gb with up to 350w for the cpu alone.

 

For performance per watt, the minis totally win this one.

But depending on the software, threadripper might be more effective.

[Freakwise]

I mean I kinda see it's good for something like render farm for blender or cluster for ai training but other than that Threadripper maybe be better at general use case

 

 

[Spindel]

Concering the power consumption 1 M1 mini running full tilt on both GPU and CPU at the same time draws around 35 W from the wall for the entire system. So power consumption wise it should be a lot less than only the Threadripper CPU running at full speed.

 

No expert in clusters but there should be a possibility to spilt up a workload that is heavily parallelized on this M1 mini cluster, just needs the right software (I'm thinking like CFD simulations that are spilt up in chunks and sent to clusters).

 

But yeah for the right kind of work loads that can be split up in this way you should actually be able to get a fairly good result at a lower power budget if you can manage to build a M1 Mini cluster. 

[Baradh]

@Kilrah indeed. But Threadripper is also a very case-specific hardware as Linus said. So it would not be too far off to compare these two ideas. But again I don't know if you can connect ARM-based Minis together (although with 10 bit ethernet and a high quality switch it is probably doable).

 

@LTTfan2006 I was thinking of network systems. That is why I included the 10Gigabit ethernet. According to Linus' video, it doesn't share any lane with other hardware so the data stream should be smooth all the time. Again, the program should be able to use that parallel structure of course.

 

@SupaKomputa In addition to the CPU cores, there are 16 neural engines and for some tasks even 8 GPU cores (like when you use CPU cores for denoising, even you are rendering with a GPU). So if the application supports them, there may even be a better performance than those. But maybe there will be an incredible bottleneck, I don't know 

 

@Freakwise Probably yeah. But for performance per buck, there is a mobility element to this idea. With a laptop, you can detatch a part of the system and take wherever you go. For Threadripper, you don't have that chance.

Overall, there are possible drawbacks (and literal, due to lack of applications in macos in some areas) but I really would love to see some practical experiments with this idea

[RollinLower]

the big thing with threadripper is that all cores are on the same substrate, and inter-die latency is much lower as compared to dividing a workload into multiple 'nodes'  in a cluster with just 10G between them, which is comparatively slow.

8 minutes ago, Baradh said:

But for performance per buck, there is a mobility element to this idea. With a laptop, you can detatch a part of the system and take wherever you go. For Threadripper, you don't have that chance.

that is an interesting idea, take a part of the cluster with you as an external compute node, but would require tailor made software or atleast a lot of custom work in your clustering program.

[Baradh]

12 minutes ago, RollinLower said:

the big thing with threadripper is that all cores are on the same substrate, and inter-die latency is much lower as compared to dividing a workload into multiple 'nodes'  in a cluster with just 10G between them, which is comparatively slow.

that is an interesting idea, take a part of the cluster with you as an external compute node, but would require tailor made software or atleast a lot of custom work in your clustering program.

My initial idea was about Blender and I don't know how Blender handles parallel computers when it comes to rendering. Maybe Blender and other modeling systems have built-in approaches to these kind of situations? Most of the movies are being rendered on farms so that means different computers connected on a grid. Threadripper is also a desktop workstation CPU, has drawbacks from Epyc and strenghts than regular Ryzen.

So I am not sure (to be honest, don't have the slightest idea) about the custom work part. It may be needed, but when you take the laptop our of the equation (not during the process, before the process of course I wouldn't dream of making on the go alterations) maybe the workload is shared evenly on 4 Mac Minis.

By the way I am using an old computer with 2600K (Sandy Bridge i7) at home but a Ryzen 2600X at work. And I use remote desktop to make changes on the model and render it using my work computer. This way I can remotely oversee my progress. For example in Blender, I render 2/3 of the frames from work and support it with 1/3 of the frames from home. So it shortens the time, albeit slightly. These kind of practices got me thinking on this situation. And also expandibility with future Mac Minis.

Edited June 2, 2021 by Baradh
lots of typos, sorry!

[Spindel]

Someone should make this and make a program that can use it (even if it will be a nonsense program with no practical use) 

 

HEY @GabenJr!

[igormp]

35 minutes ago, Freakwise said:

cluster for ai training

ML usually relies on GPUs, and a single 3060 would be way better than 10 mac minis.

[Spindel]

1 minute ago, igormp said:

ML usually relies on GPUs, and a single 3060 would be way better than 10 mac minis.

Make it a cluster for CFD computation then. 

[igormp]

4 minutes ago, Spindel said:

Make it a cluster for CFD computation then. 

Too bad star ccm+ doesn't run on MacOS (nor even ARM)

[Spindel]

4 minutes ago, igormp said:

Too bad star ccm+ doesn't run on MacOS (nor even ARM)

There always is OpenFOAM, don't know if it's any good or works with ARM tho. 

 

But this as I stated earlier would be a fun LTT video, even doing some nonsense computation just to try as a PoC. 

[mariushm]

Jesus dude, you could just get on a bus or a car and go to Romania or somewhere close to you and pick up a threadripper cpu and motherboard from a retail store. 

 

Place an order first or give a call to the store to hold them in store for pickup if needed because lots of stores list them with note "available at distributor" which means the store gets them in 1-2 days from order directly from the company which imports them in the country. 

 

It's a 10 hour drive and another 10 hour back, if you go by car. 

 

Also, there's services which redirect packages, so you could order from Amazon com or co.uk / whatever or other stores and have it shipped to a US address which they can then  forward the package to you.

[Baradh]

@mariushm this is a thought experiment. I'm not looking for a buying option I'm just trying to make a case study and would love to see someone try that practically.