render farm for the lols

[Pesky Ngon]

hi...

I do 3D stuff for a living but I also love messing around with PCs and searching for deals, building, upgrading, etc. so although having a lot of power is genuinely helpful for my 3D work, I like going overkill for fun just to see how fast I can make my system for as cheap as possible. and now I want to take things to the next level. in 3D applications, using CPUs on Linux in some situations can mean a 2x performance boost compared to windows. but I rely on windows for certain programs that simply don't exist elsewhere. so instead of spending any more money upgrading my main rig, Ive realised it might be more cost efficient from this point on to buy slightly lower spec, but more power efficient parts for pcs, and use them as nodes. my main hope is that I can get some crazy dual or quad CPU mobo from an old server, install Linux for the 1.5x performance, and then just have it connected as a node (or nodes if I buy multiple). problem is...I know f*** all about servers and I know that if I try and buy one ill mess it up and waste all my money. so I was wondering if anyone could give me some general tips about how server hardware is different from normal desktops, and where I should look to buy a used, power efficient server that has kinda midrange specs. any help would be great.

[Jarsky]

does your rendering software support clustering? 

[KarathKasun]

Just get a few $50-$100 servers with 2x quads in them.  Only turn them on when you want to render, hibernate them when you are not.

 

If you really want to get some networking skills under your belt, make them all boot off of the network diskless from a single master server.  Maybe even get some 10GigE network cards from server surplus and a 10GigE switch.

 

You can reasonably set up a decently powerful high speed cluster for less than $5k.

[Pesky Ngon]

Just now, Jarsky said:

does your rendering software support clustering? 

im afraid I don't know what that means. what I do know, is that I can split the workload between as many pcs as I want. so its just a case of getting a bunch of pcs that wont break the bank

 

[Pesky Ngon]

1 minute ago, KarathKasun said:

Just get a few $50-$100 servers with 2x quads in them.  Only turn them on when you want to render, hibernate them when you are not.

 

If you really want to get some networking skills under your belt, make them all boot off of the network diskless from a single master server.

true. im an idiot for not thinking of turning them off in not in use. thanks for reminding me

if I buy an old server, is it as simple as just plugging in a hard drive and installing Linux like you would on a normal pc, or are there extra steps I need to look out for?

[KarathKasun]

Just now, Pesky Ngon said:

true. im an idiot for not thinking of turning them off in not in use. thanks for reminding me

if I buy an old server, is it as simple as just plugging in a hard drive and installing Linux like you would on a normal pc, or are there extra steps I need to look out for?

You could in theory just get drives for one of them and diskless boot linux on them.

[Jarsky]

Just now, Pesky Ngon said:

im afraid I don't know what that means. what I do know, is that I can split the workload between as many pcs as I want. so its just a case of getting a bunch of pcs that wont break the bank

 

I mean parallel compute....from your reply it sounds like the answer is yes. After Effects can do it as an example. 

Just get some PowerEdge R710 barebones with dual X5650's. Will give you plenty of compute. 

[KarathKasun]

5 minutes ago, Jarsky said:

does your rendering software support clustering? 

Blender has plugins for clustering.

[KarathKasun]

Here is a decent starting point for what you are looking for...

 

https://www.ebay.com/itm/Dell-PowerEdge-R410-2x-Intel-Xeon-X5675-3-07GHz-16GB-DDR3-2x-250GB-HDD-1U-Server/312237096146?hash=item48b2c7fcd2:rk:1:pf:0

 

12 cores and 24 threads per box, 3ghz with 3.4ghz turbo.

They will need an additional 4x8gb of memory to maximize performance, ECC/REG memory is cheap though.

[Pesky Ngon]

7 minutes ago, KarathKasun said:

You could in theory just get drives for one of them and diskless boot linux on them.

im afraid I don't know what discless boot is, but ill look into it, thanks

[Pesky Ngon]

7 minutes ago, Jarsky said:

I mean parallel compute....from your reply it sounds like the answer is yes. After Effects can do it as an example. 

Just get some PowerEdge R710 barebones with dual X5650's. Will give you plenty of compute. 

thanks for the recommendation

 

[KarathKasun]

3 minutes ago, Pesky Ngon said:

im afraid I don't know what discless boot is, but ill look into it, thanks

Where the servers (AKA compute nodes) have no HDD.  They will load their OS/files over the network.

The box hosting the HDD's would need a to be connected with 10GigE though.

The reduced number of disks would lower power requirements by a bit.

 

https://help.ubuntu.com/community/DisklessUbuntuHowto

[Pesky Ngon]

1 minute ago, KarathKasun said:

Where the servers (AKA compute nodes) have no HDD.  They will load their OS/files over the network.

The box hosting the HDD's would need a to be connected with 10GigE though.

The reduced number of disks would lower power requirements by a bit.

ahh, I see. I think HDDs draw so little power compared to the CPUs ill be running that it will be negligible for what im doing. and the up front cost is pretty intimidating seeming as the cards for the 10 gig e would cost as much as the server. I think considering im new to this it will be safer to stick to what I know and I have some random hard drives lying around the house anyway so may as well put them to use.

[KarathKasun]

Get 10GigE cards off of ebay.  Also, you only need a switch with a single 10GigE uplink.

The former can be had for ~$80 and the latter can be had for ~$50.

 

Also, the HDD's likely need to be SAS, not SATA.  However, that server linked has 2x 250gb drives in it and there are more than one server available to buy.

[KuJoe]

1 minute ago, Pesky Ngon said:

ahh, I see. I think HDDs draw so little power compared to the CPUs ill be running that it will be negligible for what im doing. and the up front cost is pretty intimidating seeming as the cards for the 10 gig e would cost as much as the server. I think considering im new to this it will be safer to stick to what I know and I have some random hard drives lying around the house anyway so may as well put them to use.

The HDDs account for a significant amount of power in servers. If you want to save on power and heat then switching from HDDs to SSDs will make a noticeable difference. Remember, each HDD has a motor in it and motors require power to spin fast for long periods of time (even the motor in your phone to cause it to vibrate when you get a message or a call uses quite a bit of power). You can switch to 5400 RPM 2.5" drives to save some power if you don't need a lot of IO performance.

[Pesky Ngon]

Just now, KuJoe said:

The HDDs account for a significant amount of power in servers. If you want to save on power and heat then switching from HDDs to SSDs will make a noticeable difference. Remember, each HDD has a motor in it and motors require power to spin fast for long periods of time (even the motor in your phone to cause it to vibrate when you get a message or a call uses quite a bit of power). You can switch to 5400 RPM 2.5" drives to save some power if you don't need a lot of IO performance.

 

6 minutes ago, KarathKasun said:

Get 10GigE cards off of ebay.  Also, you only need a switch with a single 10GigE uplink.

The former can be had for ~$80 and the latter can be had for ~$50.

 

Also, the HDD's likely need to be SAS, not SATA.  However, that server linked has 2x 250gb drives in it and there are more than one server available to buy.

hmm. another thing im thinking about is where im likely to be living fairly soon. ill be renting a flat with friends as a roomshare. the important part is a lot of these flats offer to pay your electricity for you, in which case the HDDs power draw wont matter. I think for now I will use HDDs and then in future if I have enough servers to justify it, ill make the change to 10gig e and repurpose the HDDs. thanks a lot for the tips though.

[KarathKasun]

Best course of action, buy one or two of those servers, install Ubuntu, test how much it helps your render times with Crowd Render.  Worst case scenario, you are out ~$300 but have two decently powerful servers that you can set aside to CPU render complex scenes or animations.

 

Oh, do note that these will be LOUD.  A whole new level of LOUD that you would not necessarily associate with computers unless you have been in a server room.

[Pesky Ngon]

1 minute ago, KarathKasun said:

Best course of action, buy one or two of those servers, install Ubuntu, test how much it helps your render times with Crowd Render.  Worst case scenario, you are out ~$300 but have two decently powerful servers that you can set aside to CPU render complex scenes or animations.

tbh, even if the power efficiency ends up not being great, it'll still be a really fun project to mess around with new parts. I think it could also help me a lot for simulation work If I can dump CPU bound physics sims onto other PCs so I can keep using my main rig smoothly.

[KuJoe]

Just some stats from two identical servers for anybody interested in power usage, one with 6x SSDs and one with 6x HDDs:

 

Current Power Usage

HDDs: 217W (1.8A)

SSDs: 161W (1.4A)

 

Power Cost

HDDs: $35.81 per month

SSDs: $20.55 per month

 

Peak Power Usage

HDDs: 349W (2.9A)

SSDs: 236W (1.9A)

[Pesky Ngon]

2 minutes ago, KuJoe said:

Just some stats from two identical servers for anybody interested in power usage, one with 6x SSDs and one with 6x HDDs:

 

Current Power Usage

HDDs: 217W (1.8A)

SSDs: 161W (1.4A)

 

Power Cost

HDDs: $35.81 per month

SSDs: $20.55 per month

 

Peak Power Usage

HDDs: 349W (2.9A)

SSDs: 236W (1.9A)

thanks, that's really useful. I hadn't realised the HDDs would use that amount of power. really surprised. considering how cheap SSDs have become in the last few months I think itll be worth me picking some up instead of using the hard drives.

[leadeater]

4 hours ago, Pesky Ngon said:

hmm. another thing im thinking about is where im likely to be living fairly soon. ill be renting a flat with friends as a roomshare. the important part is a lot of these flats offer to pay your electricity for you, in which case the HDDs power draw wont matter. I think for now I will use HDDs and then in future if I have enough servers to justify it, ill make the change to 10gig e and repurpose the HDDs. thanks a lot for the tips though.

Power may or may not be an issue but noise will. The used servers on ebay are great options but they are by no means quiet, like think near vacuum cleaner (not quite but be warned).

 

Also my advice generally is keep it simple, you don't need diskless and won't benefit from it. Focus on the things you need, CPUs first then RAM followed by network which is the first point where you can cut costs. 10Gb is a lot better but if we're only talking a couple of mins to transfer files to render then not worth it, if it has to stream the data in real time while rendering then all you need is more network bandwidth than the bitrate demands.

[Pesky Ngon]

9 hours ago, leadeater said:

Power may or may not be an issue but noise will. The used servers on ebay are great options but they are by no means quiet, like think near vacuum cleaner (not quite but be warned).

 

Also my advice generally is keep it simple, you don't need diskless and won't benefit from it. Focus on the things you need, CPUs first then RAM followed by network which is the first point where you can cut costs. 10Gb is a lot better but if we're only talking a couple of mins to transfer files to render then not worth it, if it has to stream the data in real time while rendering then all you need is more network bandwidth than the bitrate demands.

yea, ive seen some servers in videos when i was researching. as far as noise goes I was thinking of buying replacement fans but im not sure if id run into compatibility issues. and the program im planning to use to transfer the files does it over wifi. as an example one of my typical files for 3D is about 50mb. but the plugin ill be using sends it in advance so its ready when I need it, so speed shouldn't be a big deal unless its really stupid slow.

[Electronics Wizardy]

42 minutes ago, Pesky Ngon said:

yea, ive seen some servers in videos when i was researching. as far as noise goes I was thinking of buying replacement fans but im not sure if id run into compatibility issues. and the program im planning to use to transfer the files does it over wifi. as an example one of my typical files for 3D is about 50mb. but the plugin ill be using sends it in advance so its ready when I need it, so speed shouldn't be a big deal unless its really stupid slow.

 Replacement fans are a pain. The best option is to put a reistor on the existing fans. But it won't be silent. These servers need the airflow to keep cool, esp when under heavy load like rendering.

 

WIfi is gonna be slow, get wired if possible.

 

 

[Pesky Ngon]

10 minutes ago, Electronics Wizardy said:

 Replacement fans are a pain. The best option is to put a reistor on the existing fans. But it won't be silent. These servers need the airflow to keep cool, esp when under heavy load like rendering.

 

WIfi is gonna be slow, get wired if possible.

 

 

what makes them a pain exactly? I know these things need to keep cool, but I wouldn't be ditching the fans, just replacing.

[Electronics Wizardy]

Just now, Pesky Ngon said:

what makes them a pain exactly? I know these things need to keep cool, but I wouldn't be ditching the fans, just replacing.

Well they have special connectors, and there normally not normal sizes(1u servers often have fans that are 40x40x56mm counter rotating.

 

Its much easier to just put reistors on the current fans to slow them down.

 

And most normal fans that you get for a gaming pc won't move enough air.