Can someone educate me on blender render farm builds?

[1bitboolean]

Budget (including currency): unlimited

Country: Latvia

Games, programs or workloads that it will be used for: Blender

I'm going pro in blender and need to build a render farm in the near future since the current 4090 PC isn't cuting it with render times. I am knowledgeable on consumer grade PCs but I'm completely blanking on enterprise. I'd like to know what is different about render farms as opposed to normal PC builds. 

The goal is to assemble a build with 2x 4090 and as time goes on keep scaling to 8-10x 4090 cards.

-What should I look for in a motherboard when buying one other than 10 PCIe slots?

-4090 doesn't support NVlink so is that a big deal, is there no other way to pool memory?

-What should I look for in a CPU (I know it has to be a threadripper but why and what stats do I look for when choosing the CPU for 10 card rig?

-What type of case do I use?

-4090 scores higher than A6000 ada however so many prebuilt renderfarms use ada even tho it's lot more expensive. Why's that?

-How much ram do I need if 64GB DDR5 kit on a single GPU rig has been more than enough? Will I need more for a farm?

-Any software to split render in a way that it renders 1 or 2 frames per 2 cards? 

-Any other software/hardware things that need to be done that normal PC doesn't require?

 

 

 

 

[Blue4130]

10 minutes ago, 1bitboolean said:

Budget (including currency): unlimited

Country: Latvia

Games, programs or workloads that it will be used for: Blender

I'm going pro in blender and need to build a render farm in the near future since the current 4090 PC isn't cuting it with render times. I am knowledgeable on consumer grade PCs but I'm completely blanking on enterprise. I'd like to know what is different about render farms as opposed to normal PC builds. 

The goal is to assemble a build with 2x 4090 and as time goes on keep scaling to 8-10x 4090 cards.

-What should I look for in a motherboard when buying one other than 10 PCIe slots?

-4090 doesn't support NVlink so is that a big deal, is there no other way to pool memory?

-What should I look for in a CPU (I know it has to be a threadripper but why and what stats do I look for when choosing the CPU for 10 card rig?

-What type of case do I use?

-4090 scores higher than A6000 ada however so many prebuilt renderfarms use ada even tho it's lot more expensive. Why's that?

-How much ram do I need if 64GB DDR5 kit on a single GPU rig has been more than enough? Will I need more for a farm?

-Any software to split render in a way that it renders 1 or 2 frames per 2 cards? 

-Any other software/hardware things that need to be done that normal PC doesn't require?

 

 

 

 

A render farm is not a single computer. It is a cluster of computers that pool resources and are all linked together using clustering software. What you are looking to build is a workstation.

 

You aren't going to get 10 4090's in a single machine. At least not easily. If you use a dual epyc cpu motherboard I guess theoretically you could. But you would need to have some cards connected through the oculink ports and others would need to be connected through riser cards to fit. And you would need multiple PSU's to power it all. It would be jank to say the least.

 

As for why people use A6000's, it can come down to the fact that they need the studio software rather than commercial nvidia drivers. It also has double the ram of the 4090.

[1bitboolean]

On 11/3/2024 at 2:01 AM, Blue4130 said:

A render farm is not a single computer. It is a cluster of computers that pool resources and are all linked together using clustering software. What you are looking to build is a workstation.

 

You aren't going to get 10 4090's in a single machine. At least not easily. If you use a dual epyc cpu motherboard I guess theoretically you could. But you would need to have some cards connected through the oculink ports and others would need to be connected through riser cards to fit. And you would need multiple PSU's to power it all. It would be jank to say the least.

 

As for why people use A6000's, it can come down to the fact that they need the studio software rather than commercial nvidia drivers. It also has double the ram of the 4090.

Interesting, I will look into clusters.

I think I can fit 10 4090s with extension cords on some rig like crypto miner bros do it. Multiple PSUs is a given and it definately wouldn't fit in a normal case anyways. Is there no way to pool 4090 memory? Because going for ada seams a monumental waste of money otherwise.

[Kilrah]

11 hours ago, 1bitboolean said:

like crypto miner bros do it

Mining needs no CPU-GPU bandwidth so typically their cards are connected via PCIe2 x1. For rendering you're gonna need the full bandwidth.

 

You're looking at 2-4 EPYC/TR systems if you want to do that well.

 

On 11/3/2024 at 1:01 AM, Blue4130 said:

As for why people use A6000's, it can come down to the fact that they need the studio software rather than commercial nvidia drivers. It also has double the ram of the 4090.

Also they're 2-slot and designed to be packed in rack-mounted servers. Consumer 3-4 slot cards will cover precious slots unless you do some jank with risers and a custom frame. Then the question is if you're gonna spend $20K on a system do you want it to be jank and potentially fail renders because of weird communication issues... maybe multiple systems with 2-3 directly mounted cards in each make more sense.

 

Or depending on the amount of work you get maybe look into rendering services, because a $20-40K investment would probably only make sense if it's in constant full use. And obviously consider the energy costs.

 

 

 

[1bitboolean]

9 hours ago, Kilrah said:

Mining needs no CPU-GPU bandwidth so typically their cards are connected via PCIe2 x1. For rendering you're gonna need the full bandwidth.

 

You're looking at 2-4 EPYC/TR systems if you want to do that well.

 

Also they're 2-slot and designed to be packed in rack-mounted servers. Consumer 3-4 slot cards will cover precious slots unless you do some jank with risers and a custom frame. Then the question is if you're gonna spend $20K on a system do you want it to be jank and potentially fail renders because of weird communication issues... maybe multiple systems with 2-3 directly mounted cards in each make more sense.

 

Or depending on the amount of work you get maybe look into rendering services, because a $20-40K investment would probably only make sense if it's in constant full use. And obviously consider the energy costs.

 

 

 

-I looked into render farm services and for most of the renders I produce they were asking 80-100$ per job (I did test jobs) so 30-50k seams ok (resale/repurpose to multiple gaming rigs is possible in the future) I render nearly every day on 2 computers (4090 and 4060) for 6-12h so seams appropriate. 
-It seams ada, multiple MBs and a server rack might be the way to go
-Could you explain why EPYC not threadrippers?  What do I need in terms of processor for a farm, what stats do I need to take into account?

[Blue4130]

4 minutes ago, 1bitboolean said:

-I looked into render farm services and for most of the renders I produce they were asking 80-100$ per job (I did test jobs) so 30-50k seams ok (resale/repurpose to multiple gaming rigs is possible in the future) I render nearly every day on 2 computers (4090 and 4060) for 6-12h so seams appropriate. 
-It seams ada, multiple MBs and a server rack might be the way to go
-Could you explain why EPYC not threadrippers?  What do I need in terms of processor for a farm, what stats do I need to take into account?

Threadripper Pro would also work. Make sure it's the Pro though, not the standard. You will need all of the lanes that the pro offers.

 

Curious though, what are you doing that needs all of this render power? Must be a business?

[1bitboolean]

27 minutes ago, Blue4130 said:

Threadripper Pro would also work. Make sure it's the Pro though, not the standard. You will need all of the lanes that the pro offers.

 

Curious though, what are you doing that needs all of this render power? Must be a business?

I joined some rich mofos on their business ventures as a designer and they have lot of 2D and vector animators but 3D.... Well I am the 3D department currently and yeah I do 1-2 scenes of animation per day and that requires hela render power especially because the syle is realism and I often find myself working with 4K textures and volumetrics not to mention the amount of polys...

[Kilrah]

9 hours ago, 1bitboolean said:

Could you explain why EPYC not threadrippers? 

TR Pro's been so niche and obscure there's almost nothing around for it, EPYC's plenty in the server world so more options and also a pretty good used market if you go previous gen or 2, it's probably the cheapest way to get plenty of PCIe.

 

9 hours ago, 1bitboolean said:

What do I need in terms of processor for a farm, what stats do I need to take into account?

That I don't know.