Build Idea - Watercooled mega plex server

[Blindsay]

Hey All,

 

First post here, followed the youtube channel for quite some time and some of my crazy build ideas have definitely stemmed from there. That said this is a build I am thinking of and was curious what you all thought



Background: Back in 2011 i won the December 2011 guru3d rig of the month award which i know many have done but i was super excited about it and have been thinking about trying to do another build to make another run at winning but i really dont game that much these days (I have a laptop with an i7 7700hq/GTX 1070) but what i do use a lot is my home server. I run unraid on it as a base and Plex runs in a VM on it. So that got me thinking, it might be cool to do a high end watercooled server as I enjoy watercooling, would like to make a silent (as possible) server and build something fun and would go to something i use a lot.


Current hardware:
Asus Z9PE-D8 WS
2x Xeon E5-2670 (v1s)
32GB (8x 4GB DDR 3 1333 ECC)
5x WD RED 8TB
1x Toshiba X300 5TB (scratch drive outside of the unraid array)
2x Samsung 850 EVO 250GB (for the VM's)

Hardware to buy:
2x Xeon E5-2680V2 (Fastest reasonably priced cpus i can put in this board, once i build it, i want to run it till it dies)
Replace the 32GB with 64GB (and maybe bump up to 1600)
Leaning towards Caselabs for the case, Maybe the Mercury S8 or something bigger. Ideally want to be able to fit ~8 3.5" drives (and a couple 2.5")
All the watercooling gear (what i get will largely depend on the case i get)
A couple SAS cards (non-raid) - Want to get some so i can run SAS to SATA breakout cables, and the onboard Intel SATA ports no longer work on my Asus board

Misc thoughts:
As with all my previous builds i will have to come up with a theme of some sort
Not sure if there are any better deals in terms of raw cpu power than the E5-2680V2's ?
Plex server will need to be beefy enough for 5-10 1080p streams at the same time, also H265 content abuses cpus. Also will be running a minecraft server on it, an exchange server and a few other small lab VM's
Are there any cases big enough to run 2 builds? IE this on 1 side and a second machine on the otherside? like if i did decide to add a gaming machine down the road

Thoughts?

[onehentaiboi]

Do not go water cooling cause it can cause leak and idea is great but water cooling is big no in server world.

[Guest]

wait theres competitions for who has the coolest pc?

my mini itx socket 478 inverted sleeper just got put on a higher priority, lel

[LienusLateTips]

26 minutes ago, onehentaiboi said:

Do not go water cooling cause it can cause leak and idea is great but water cooling is big no in server world.

Water cooling is probably fine if you don't move it. 

 

 

Sounds like a... ok idea.

[Dark]

3 hours ago, Blindsay said:

Hardware to buy:
2x Xeon E5-2680V2 (Fastest reasonably priced cpus i can put in this board, once i build it, i want to run it till it dies)
Replace the 32GB with 64GB (and maybe bump up to 1600)
Leaning towards Caselabs for the case, Maybe the Mercury S8 or something bigger. Ideally want to be able to fit ~8 3.5" drives (and a couple 2.5")
All the watercooling gear (what i get will largely depend on the case i get)
A couple SAS cards (non-raid) - Want to get some so i can run SAS to SATA breakout cables, and the onboard Intel SATA ports no longer work on my Asus board

Misc thoughts:
As with all my previous builds i will have to come up with a theme of some sort
Not sure if there are any better deals in terms of raw cpu power than the E5-2680V2's ?
Plex server will need to be beefy enough for 5-10 1080p streams at the same time, also H265 content abuses cpus. Also will be running a minecraft server on it, an exchange server and a few other small lab VM's
Are there any cases big enough to run 2 builds? IE this on 1 side and a second machine on the otherside? like if i did decide to add a gaming machine down the road

Thoughts?

Non-raid SAS cards = HBA (host bus adapter)

 

E5-2687W v2 would be better CPU, but not a comparison for cost.

 

If you have a lot of H265 content then it may be more beneficial to run a more current generation processor with QuickSync support.

 

Dual home cases exist but the premium usually outweighs the benefits (the money would be better spent on 2687w cpus IMO).

[Blindsay]

4 hours ago, onehentaiboi said:

Do not go water cooling cause it can cause leak and idea is great but water cooling is big no in server world.

Thanks but i have done quite a few watercooled builds so i am not worried about leaks. Also this is a home server, not some critical production server

3 hours ago, JDE said:

Water cooling is probably fine if you don't move it. 

 

 

Sounds like a... ok idea.

Yeah its a home server so it wont be moving. I like watercooling but i dont do a lot of gaming so a watercooled server seemed like a fun project

39 minutes ago, Dark said:

Non-raid SAS cards = HBA (host bus adapter)

 

E5-2687W v2 would be better CPU, but not a comparison for cost.

 

If you have a lot of H265 content then it may be more beneficial to run a more current generation processor with QuickSync support.

 

Dual home cases exist but the premium usually outweighs the benefits (the money would be better spent on 2687w cpus IMO).

Yeah the clockspeed on those is nice but they are definitely a good chunk of change more than the 2680V2. I can get a pair of the 2680V2's for like $436

I dont have a ton of H265 content at the moment but its becoming harder to avoid.

Ill stick to just a case for the server then. Still considering caselabs but i was also checking out the 900D, Tower 900 and there are a couple other Thermaltake cases that seem to fit the build

Thanks!

[Salv8 (sam)]

On 3/5/2018 at 3:10 AM, onehentaiboi said:

Do not go water cooling cause it can cause leak and idea is great but water cooling is big no in server world.

i have a dude try this, they lost a $10,000 AUD server because a leek...

never do it unless you are google as they water cool their servers but they know what they are doing...

[leadeater]

22 hours ago, Salv8 (sam) said:

i have a dude try this, they lost a $10,000 AUD server because a leek...

never do it unless you are google as they water cool their servers but they know what they are doing...

 

On 3/5/2018 at 6:10 AM, onehentaiboi said:

Do not go water cooling cause it can cause leak and idea is great but water cooling is big no in server world.

Water cooled servers is not uncommon on the very high end server deployments, HPE SGI comes to mind as a good example. There's also two kinds of server water cooling: Direct to Chip and Rack. Direct to chip is rather obvious but Rack is where there is a heat transfer unit installed in the back of a rack, basically a radiator, that collects the exhaust heat from all the servers and is transferred by water to another room or to the roof of the building where the heat is dissipated by radiators there. Some installations also use chilled water.

 

Weta Digital is a local example for me of someone doing Direct Chip and Rack server water cooling with HPE Apollo servers.

https://cc.cnetcontent.com/vcs/hp-ent/inline-content/TN/8/F/8F8390E2B76346D1D1E4FDD5910ABC99A9682793_source.PDF

[Guest]

For this sort of build I don’t get the argument of ‘don’t do it, it’ll leak’. Appart from a few model numbers, this sort of thing is no different from a desktop PC being done. 

 

Howver, you need to keep in mind things like maintainence and how long that’ll take, will there be any issue in it being off for a few hours to clean it out every 6-12 months?

[Mikensan]

With your previous water cooling builds I suspect you'll well versed enough to know what you're getting into. (Anyone complaining about servers is quickly forgetting this will be sitting right next to you, not unattended in a datacenter).

 

I would be sure to focus on storage, movies take up so much darn space! Make sure you have enough slots for future growth. You could also alternatively get whatever case you want, use a HBA card with an external connector and have a separate shelf for storage.

 

For internal HBA cards if you want an all in one solution, a 9211 from ebay would give you the ability to connect 8 disks (2x SFF-8087 to SATA). Usually about $60.

 

Virtualization is addictive, so I support you getting more RAM. I only have 128gb in my lab, and I'm looking to buy more.

 

Now it's not the most beautiful, but going as fanless as possible is awesome imo. But if that CPU is going to get pegged for hours at a time it might be best not to.

 

Do you have a link to the post or some pictures from when you won? 

[Maslygan]

If youre going to transcode h265, then perhaps go with a pascal GPU aswell. It can only encode 2 streams, because of Nvidia software limitations, but it will give it a good boost.
actually you can just use any card above 700 series, but pascal offers 10bit 4k encoding with all sorts of goodies, and h265 aswell iirc.

 

watch this video from byte my bits, it's interresting: 

 

[leadeater]

5 hours ago, Maslygan said:

If youre going to transcode h265, then perhaps go with a pascal GPU aswell. It can only encode 2 streams, because of Nvidia software limitations, but it will give it a good boost.
actually you can just use any card above 700 series, but pascal offers 10bit 4k encoding with all sorts of goodies, and h265 aswell iirc.

 

watch this video from byte my bits, it's interresting: 

 

Do you know what the limit is on AMD GPUs? I had a quick look and couldn't find anything on max encode streams.

[Salv8 (sam)]

On 09/03/2018 at 11:34 AM, leadeater said:

-snip-

a. they are pre designed and tested so they are sure that they are not going to leek

b. they are expensive and unless you are doing workloads that require a water cooled server, it's not worth it,

thats what i was saying, it's not worth it unless you need it

[leadeater]

19 minutes ago, Salv8 (sam) said:

a. they are pre designed and tested so they are sure that they are not going to leek

Nothing is stopping you from doing this testing and ensuring good workmanship, all my personal servers at home I've built are water cooled and have been for years. The only server of mine not is an IBM x3500 M4 which I've waiting for the warranty to run out before touching it.

 

Pre-designed and tested doesn't save you from failures and leaks, hell at work we had a CPU heatsink come with a manufacturing fault in a HPE DL580 Gen9. The copper cold plate came loose from the rest of the heat sink body and it was not providing any cooling at all, not great with a $10,000 CPU under it.

 

19 minutes ago, Salv8 (sam) said:

b. they are expensive and unless you are doing workloads that require a water cooled server, it's not worth it,

thats what i was saying, it's not worth it unless you need it

It's worth it at home if you don't want 3 vacuum cleaners running all day 24/7.

[Salv8 (sam)]

1 minute ago, leadeater said:

It's worth it at home if you don't want 3 vacuum cleaners running all day 24/7.

my server costs $10 AUD, like hell i'm water cooling the thing, i would rather have that thing sound like the jet engine screams from hell then have the system damaged by a leek

[leadeater]

27 minutes ago, Salv8 (sam) said:

my server costs $10 AUD, like hell i'm water cooling the thing, i would rather have that thing sound like the jet engine screams from hell then have the system damaged by a leek

If you don't want to then don't. But there is no more risk of leaks than in a normal desktop and those have been successfully water cooled very easily even by novices. If someone is experienced in building water cooled systems, the OP is, I don't see any justification for saying don't do it. Your original reasoning was only do it if you know what you are doing, well this person does. You don't have to be Google to know how to do it or have to buy a tailored system from HPE or IBM etc to do it.

 

Personally I would rather not have my gaming desktop killed by water than any of my servers and that costs less yet I still did a completely custom job of it water cooling everything down to the ram because why not, I wanted to do it and no one saying that it's dumb or not worth it is going to stop me doing it. I'd rather people just give advice and help than get told not to do it, I already know water cooling ram is asinine.

 

The OP wants to build a custom quiet server, well the best way that is achieved is by water cooling if it's a powerful dual socket server. Smaller single socket lower power servers do just fine with a decent air cooler and a good fan and good case airflow but the heat generated by two CPUs requires higher RPM fan settings on the case fans making it rather loud.

 

I've lived through loud an obnoxious servers and disk arrays in or near my bedroom, never again.

 

I'd be far more worried about bad power killing the server than water, exactly why I have an Eaton 9130 UPS protecting them.

[Maslygan]

5 hours ago, leadeater said:

Do you know what the limit is on AMD GPUs? I had a quick look and couldn't find anything on max encode streams.

Honestly dude, i have no idea. i don't know if you saw the video i posted, as it seems to be removed now in the forums for some reason, and replaced with an add for amazon lol. But im pretty sure that he talked about it. I don't actually have any amd gpu's to test it with, so i cannot tell you the answer. But most likely there is no limit, and something like a rx570, i would assume is pretty good for that, but im not sure. If they even allow encoding on amd that is.

[Guest]

On 3/10/2018 at 11:22 AM, Maslygan said:

watch this video from byte my bits, it's interresting: 

 

Really cool video thanks! - Think I should have a 750Ti around somewhere. Been struggling with 4-6 different people trying to watch content off an E3 for a while.

[GDRRiley]

On 3/8/2018 at 5:34 PM, leadeater said:

Direct to chip is rather obvious but Rack is where there is a heat transfer unit installed in the back of a rack, basically a radiator, that collects the exhaust heat from all the servers and is transferred by water to another room or to the roof of the building where the heat is dissipated by radiators there. Some installations also use chilled water.

I just went to a super computer at Berkeley Laurence Labs and they have airflow going sideways with rads in between racks but still same concept. 

 

[leadeater]

17 minutes ago, GDRRiley said:

I just went to a super computer at Berkeley Laurence Labs and they have airflow going sideways with rads in between racks but still same concept. 

Ooo that would have been cool, so jealous 

[Blindsay]

On 3/9/2018 at 1:20 PM, Mikensan said:

With your previous water cooling builds I suspect you'll well versed enough to know what you're getting into. (Anyone complaining about servers is quickly forgetting this will be sitting right next to you, not unattended in a datacenter).

 

I would be sure to focus on storage, movies take up so much darn space! Make sure you have enough slots for future growth. You could also alternatively get whatever case you want, use a HBA card with an external connector and have a separate shelf for storage.

 

For internal HBA cards if you want an all in one solution, a 9211 from ebay would give you the ability to connect 8 disks (2x SFF-8087 to SATA). Usually about $60.

 

Virtualization is addictive, so I support you getting more RAM. I only have 128gb in my lab, and I'm looking to buy more.

 

Now it's not the most beautiful, but going as fanless as possible is awesome imo. But if that CPU is going to get pegged for hours at a time it might be best not to.

 

Do you have a link to the post or some pictures from when you won? 

Sure, here is the post with the article and has all of the completed photos - http://www.guru3d.com/articles-pages/guru3d-rig-of-the-month-december-2011,1.html

Still trying to figure out what case i want to use for the build, for me that it sets the tone of the entire build so im usually pretty slow to choose, especially when cases of this size are not exactly cheap lol.