[Help] Cooling and PSU Advice for Server

[Lohita]

Hey Everyone, I'm new here, but have been avidly watching LTT videos daily for the last 4 or so years. I've always wanted to put together my own server at home (ever since watching this video), and have finally made a start on it. I've just purchased a 4U server case with 8 hot swap bays, and am soon going to be getting a Supermicro X9DRL-iF Dual Socket R LGA 2011 Server Motherboard, and 2 x Intel Xeon e5-2650 v2. I also plan to run 4 WD Red 2 TB HDDs for now with unRAID, and 2 SSDs in RAID to use as a cache drive, and potentially a GTX 1030/1060 for some 4k video transcoding. The plan is to have a NAS for local photo, video, and document backups, as well as run a few VMs.

 

I need your advice on a few areas:

• What PSU should I go for? From my calculations a 650W should suffice, and provide enough headroom for future HDD expansion. Or should I go for more?
• What CPU coolers should I go for?

Any help is most appreciated! Thanks!

[heimdali]

I'd get an 850W Titanium one from Seasonic.  It goes like this: 8 disks make for 100W, the CPUs make for another 200, a graphics card my need 350 and then you may HBAs or RAID cards which can draw another 20W, plus a 10GB network that may be rated for 15W and a bunch of fans (like 7 or 8 depending on your case and another fan blowing on the cards) at 5W each is another 40W.  You could get away with 750W but I wouldn't try that.  Of course, that's like the worst case, but you get the idea.

 

The WD Red are utterly slow and I don't recomment them.  You might find 2TB  HGST Ultrastar (not Deskstar!) (even used on ebay).

 

You could go for a Noctua NH-U12DX i4 if you need it to be quiet.  Dynatron makes nice coolers, too: https://www.dynatron.co/air-cooler?APPLICATION=3U%26UP&CPU Socket=LGA2011%2F2066%2FNARROW|LGA2011%2F2066%2FSQUARE

 

Which case did you get and does it come with fans?

 

[Lohita]

10 hours ago, heimdali said:

I'd get an 850W Titanium one from Seasonic.  It goes like this: 8 disks make for 100W, the CPUs make for another 200, a graphics card my need 350 and then you may HBAs or RAID cards which can draw another 20W, plus a 10GB network that may be rated for 15W and a bunch of fans (like 7 or 8 depending on your case and another fan blowing on the cards) at 5W each is another 40W.  You could get away with 750W but I wouldn't try that.  Of course, that's like the worst case, but you get the idea.

Thanks Heimdali, that makes a lot of sense. I'll go with an 850W then.

 

10 hours ago, heimdali said:

The WD Red are utterly slow and I don't recomment them.  You might find 2TB  HGST Ultrastar (not Deskstar!) (even used on ebay).

Yeah, I thought that too, but given this is for NAS mainly, and I'll have the SSD cache, will it be as much of an issue? Also, with the cache, will 2 480 GB SSDs suffice (they will be in RAID 1), or should I go for 1 TB SSDs?

 

10 hours ago, heimdali said:

You could go for a Noctua NH-U12DX i4 if you need it to be quiet.  Dynatron makes nice coolers, too: https://www.dynatron.co/air-cooler?APPLICATION=3U%26UP&CPU Socket=LGA2011%2F2066%2FNARROW|LGA2011%2F2066%2FSQUARE

Yes, I'd prefer it be quiet, so will look at the Noctua. What about the Be Quiet Dark Rock 4 or Dark Rock 4 slim?

 

10 hours ago, heimdali said:

Which case did you get and does it come with fans?

The case is a Guanghsing 4U 19" Rackmount ATX Server Chassis. It comes with 5 fans (3 x 90 mm in the front, and 2 x 60 mmm in the back). Depending on how the rear fans are, I might replace them with quieter ones later on.

[heimdali]

The WD Reds are slow and don't get any faster when you put them into a NAS device.  I have 4 of them in a server at work with ZFS running Gentoo.  The server had a Samsung Pro SSD for cache until the SSD wore out.  It seems to run faster now without the SSD, so I didn't put one back in.

 

If you can get the WD Reds for a particularly good price and don't mind them being slow, they are ok if you can return them under warranty.  IIRC, out of 8 at work, two failed under warranty.  That's a 25% failure rate.

 

I've never used UnRaid, so I can't tell what difference more or less SSD cache makes.  I'd be reluctant to use SSD for cache because SSDs wear out and because they are so expensive.  Which of the SSD manufactureres gets their firmware right?  Samsung apparently doesn't, unless they finally improved.  Intel seems to have it right.

 

I think you probably don't need SSD cache.  If you don't get 100MB/sec over 1GB network from 4 WD Reds without SSD cache, then something is wrong.  The SSD cache might with help latency, but if you need that, you should use 10GB.

 

As to ZFS, I would avoid it unless there was a very particular use case.  I've had it at home with 6 disks (1TB HGST Ultrastars) in my workstation, and the performance sucked.  ZFS also doesn't go well with Linux due to licensing issues.  You could use FreeBSD, but FreeBSD has a bad implementation of NFS which requires you to export a whole file system with given permissions rather than being able to export each folder with individual permissions for each client.  Maybe UnRaid does all this better, though I doubt that a small NAS with a couple disks is a good use case for ZFS.

 

Btrfs has recently become the standard file system for Fedora and I'm going to use it.  It has great features and I like it much better than ZFS.  I'll have to see how the performance turns out.

 

Other than that I have yet to see a machine that isn't being bogged down by software raid.  Despite hardware RAID controllers having their disadvantages, there is a lot to be said for them.  If you want cost-efficient hardware RAID, you could use a HP smart array P410 with a 1GB RAM card and a capacitor as backup battery --- but only if the firmware has been updated to a version that works with disks over 2TB.  Updating that firmware may not be so easy.  But then, since you're getting 2TB disks, it would work fine.  Updated P410s do work with WD Red 4TB disks, so 2TB should work as well.  HP provides RPM packages with software for them.  One P410 can connect 8 disks, so you'd need only one of them for your case.

 

If your mainboard has 10 SATA connectors, just use those.  You'd have 8 for the 8 disks and 2 from boot from SSDs.  Does UnRaid boot from RAID?

 

The Be Quiet coolers seem to be the same price as the Noctua ones.  For same price I would go for Noctua.  I don't have any of thier coolers but a couple of their fans, and the quality is outstanding.

 

I think I forgot to mention that you need to check the height of the cooler.  It would suck if you put on a cooler just to find out that you can't put the cover on your case because the cooler is too high.

 

The case you picked looks good   I guess the 3 fans not visible on the pictures are for cooling the disks --- at least they should.  You're gone have fun to put it all together

 

[Lohita]

Thanks, that's a lot of great info! I'm going to go with the HGST DC HA210 2 TB HDDs based on your recommendation. I've also decided to go for Noctua NH-U9DX i4 coolers - they will fit a 4U case, as the larger NH-U12DX i4 unfortunately won't. The plan is to run Btrfs on the SSD cache, so I'm glad you've recommended it too.

 

10 hours ago, heimdali said:

The case you picked looks good   I guess the 3 fans not visible on the pictures are for cooling the disks --- at least they should.  You're gone have fun to put it all together

Thank you! Yes, that's right, they're for cooling the disks, and yes, definitely going to have fun pulling it all together :). Hopefully this should all come together nicely!

[heimdali]

The Ultrastars look like fine disks.  It seems they even made a new version.

 

How do you plan on using the SSDs as cache when you use btrfs?  Using disks for cache is something that ZFS (and some hardware RAID cards can use SSDs as cache) can do.  Btrfs doesn't.  I guess you could have btrfs put the metada on the SSDs, but I wouldn't mess with that.

 

Are you gona post pictures?

 

[Lohita]

18 minutes ago, heimdali said:

How do you plan on using the SSDs as cache when you use btrfs?  Using disks for cache is something that ZFS (and some hardware RAID cards can use SSDs as cache) can do.  Btrfs doesn't.  I guess you could have btrfs put the metada on the SSDs, but I wouldn't mess with that.

From what I understand, unRAID allows you to use a cache pool (two SSDs in RAID 1) and then use btrfs on top of that.

 

20 minutes ago, heimdali said:

Are you gona post pictures?

Yes! Once I'm done, I hopefully will

 

Btw, do you have any recommendations for a UPS for this config? I could go for one later, but just wondering if perhaps I should go for it sooner rather, than later.

[heimdali]

I thought UnRaid uses the caches of ZFS?

 

For an UPS, get a 19" from APC.  They make good UPSs which last like forever and you'll probably never have issues getting replacement batteries.  With an 850W PSU, you'll probably need at least a 1500VA model.  If you have more things to connect to it, you may need to go for a larger one.  APC has runtime graphs on their website ...

 

They're expensive.  If you can find a used one on ebay and the batteries have been replaced, you can save a lot of money.  You can always check what a new battery pack would cost you for a particular model in case you find a used one for a good price that doesn't have new batteries.  They are also heavy, so if you're contemplating getting a larger one like 3000VA, you need to figure out how to handle that kind of weight beforehand

 

If you're lucky, you can even find new old stock.  Mine was about 7 years old when I bought it; the batteries were replaced and it's as good as new, just a lot cheaper (like 1/3 of normal price).

 

[Lohita]

5 hours ago, heimdali said:

I thought UnRaid uses the caches of ZFS?

Apparently the newer versions support btrfs. 

 

Thanks for the UPS info - I'll have a look at a used APC one! If I go for a 19" (rack mount I presume), do I need a PDU? 

[Jarsky]

On 10/5/2020 at 2:18 AM, Lohita said:

• What PSU should I go for? From my calculations a 650W should suffice, and provide enough headroom for future HDD expansion. Or should I go for more?

I have a Seasonic 80+ Gold 650w on my old server with 2 x X5650's, 14 HDD's and 2 SSD's. A reputable 650w will be just fine for your use case. 

On 10/5/2020 at 2:18 AM, Lohita said:

• What CPU coolers should I go for?

I've found Dynatron to be very capable for cooling server CPU's, they offer slim profile coolers, and they use ball bearing fans which are reliable and quiet. 

[Jarsky]

6 hours ago, heimdali said:

I thought UnRaid uses the caches of ZFS?

UnRAID doesn't use ZFS natively. It uses BTRFS for the cache, and has its own proprietary array implementation using XFS. 

From discussions in the pre-release build forums, theyre looking to introduce multiple array support (they just introduced multiple cache pool support in 6.9.0), and then eventually ZFS support, which looks like it won't be until the next major release (7.0)

[heimdali]

2 hours ago, Lohita said:

Apparently the newer versions support btrfs. 

 

Thanks for the UPS info - I'll have a look at a used APC one! If I go for a 19" (rack mount I presume), do I need a PDU? 

You don't need a PDU, though they can be really useful in that they can help with the mess of cables.  Instead of a pillar-mounted one, you could use something like this and zip-tie it to a pillar: https://www.brennenstuhl.com/de-DE/produkte/steckdosenleisten/ecolor-steckdosenleiste-6-fach-rot-schwarz-1-5m-h05vv-f-3g1-5  That's a lot cheaper than a PDU.

 

I don't know if you can get anything from this manufacturer; if you can, go for them because they make good quality.

 

But then, you can get PDUs that are switchable like an APC 7920 (see videos on youtube) or one from servertech.com.  The prices for those are forbiddingly high, so you have to get lucky on ebay where you can find them for under $100.

 

Those can conveniently be switched via SNMP.  I found that I'm _not_ using them to switch power in the rack.  There isn't any point in doing that because when I switch off my server, I can't very well switch the PDU anymore because of the way I have things set up, and I just don't need to do that.  A really good use case for those is when you have a number of machines that need to be started in a given order, like starting the file server and a couple minutes later the database server before starting the VM server which mounts its storage via NFS and runs VMs that run stuff that uses the database server and/or the storage.

 

So I use PDUs for switching the lights in the room, using my phone via asterisk, and with 'at' (https://linux.die.net/man/1/at).  Doing the programming for that was easy to do.

 

If you can get them for under $100, such a PDU is the most cost-effective solution for home automation I could find.  The Server Technologies one even has a web interface and counters for how much power has been going through an outlet and fancy stuff like that.  You can also get sensors for temperature and humidity (if you can find one).  They're kinda overkill, but they're hard to beat

 

[heimdali]

2 hours ago, Jarsky said:

I have a Seasonic 80+ Gold 650w on my old server with 2 x X5650's, 14 HDD's and 2 SSD's. A reputable 650w will be just fine for your use case. 

I've found Dynatron to be very capable for cooling server CPU's, they offer slim profile coolers, and they use ball bearing fans which are reliable and quiet. 

The OP wants to be prepared for video transcoding with a graphics card.  Considering that, isn't it somewhat risky to go with 650W?

 

I have a cooler from Dynatron that came with a fan on it in my server.  It's been running for 3 years or so and still runs fine.  I like the coolers they make.

 

[Jarsky]

3 hours ago, heimdali said:

The OP wants to be prepared for video transcoding with a graphics card.  Considering that, isn't it somewhat risky to go with 650W?

Ah i missed that part, then yeah i'd get a 750W for supporting an average GPU (GTX1050 Ti - GTX1660)

[Lohita]

Thanks guys!

 

@heimdali I think I may just go for a UPS a bit later when I can, and avoid the PDU as I don't expect to need one anytime soon. Just waiting on the parts to arrive and I can start putting all of this together!

[heimdali]

Sure, you can always wait until a good offer shows up and save a lot of money on an UPS that way.  Good luck with putting all the parts together!

 

[Lohita]

Thanks @heimdali! The parts are slowly arriving and I can't wait to get it all going!

 

Had another related question - I'm going to be running a Windows VM with Blue Iris as an NVR + a Deepquest AI to help with motion detection, similar to this.

The output from the cameras will be in terms of a 24x7 lower resolution 480p stream, images for processing by the AI to detect motion, and 4K clips for whenever motion is detected. Having a GPU will help with the transcoding for this. I want to store all of the output in a share on unRAID, which brings me to my question - should I turn on caching for this share or not?