passive cooling [Finished] The number cruncher: Triple Xeon passive mineral oil cooling [Update 12: Final pictures and summary]

[Stefan1024]

 

The number cruncher

 

73 cores and a Titan X in oil submerged and passiv cooled

 

 

 

Yes, my friend, it is as exorbitant as it sounds!

My last build was highly focused on single thread performance and it works perfect for this workloads: Link here.

However, I came across situations where I needed a considerable amount of compute power. As these workloads can be parallelized without a significant penalty, I developed a need for this number cruncher. Also I will use if for F@H during the night.
My first attempt was to use two E5-26xx v3 Xeons with the Z10PE-D8 WS motherboard. But it turned out to be way to expensive. Therefor I downgraded to two E5-2680 v1 Xeons on a Z9PE-D8 WS motherboard. As I saved a lot of money on the CPUs and the memory, I was able to include a Xeon Phi 31S1P. MATLAB is able to take advantage of the “automatic offload” feature to seamlessly integrate the Xeon Phi and boost the performance. I won’t write programs for the Phi by myself.
For the GPUs I will use one or two of the upcoming Polaris / Pascal line-up depending on the performance and price. I’m tending towards Polaris, but I have not enough information to decide.

 

This leads me to the part list:
CPU: 2x E5-2680 (8 cores, 2.7 GHz base clock, 3.5 GHz boost clock, 130 watts): used from Ebay
Coprocessor: Xeon Phi 31S1P (57 cores, 1.1 GHz, 8 GByte RAM, 270 watts): new from Ebay
RAM: 8x Hynix HMT31GR7CFR4C (8 GByte, DDR 3, 1600 MHz, ECC): used from Ebay
Motherboard: Asus Z9PE-D8 WS: new from retail (no used once available)
PSU: Corsair RM1000i: new from retail (I don't dare to use a used PSU)
GPU: GTX Titan X: when the GTX 1080 hype peaked, you got them for very low price, I couldnt't resist to buy one
SSD 1: Samsung 850 Evo 1TB: salvaged from my first build
SSD 2: Samsung 850 Evo 2TB: used

SSD 3: Samsung SM951 NVMe, 256 GByte: used

HDD1: WB Red 8 TB: when the helium can't escape, the oil can't come in right?
WLAN: TP-Link Archer T9E: new from retail

 

As you probably know, I have an addiction to passive cooling. This time around I tried to get rid of the heat pipes. They are expensive and vulnerable. Also I'm lacking the appropriate tools to make a CPU block out of copper. Mounting the GPUs directly to the heat sinks proved itself as a good solution. Sadly, this is not feasible for the CPUs.
So I will use two Noctua NH-U12DX i4 cooler. But now I have to get rid of the fan somehow. This is certainly possible if you look at the definition of fan:

 

 

Quote

• fan noun [C] (OBJECT/PIECE OF EQUIPMENT)

B1: an electric device with blades that turn quickly, used to move the air around.
B2: an object made of folded paper or other material that you wave with your hand in order to move the air around

 

Source : http://dictionary.cambridge.org/dictionary/english/fan

 

So be filling the box up with mineral oil, I’m not only getting rid of the noise. There are no fans left in the build by applying this definition

However, the build is not completely passive cooled, as the cooling used needs about 3 watts. But considering the total power this is negligible. So I will call it passive cooling. Also the oil itself is passive cooled.

 

More renderings:

Spoiler

 

 

 

 

 

Update 1: Super loud coil whine

Spoiler

I got the MoBo and the RAM yesterday. The first thing I did was a benchmark: 2100 in Cinebench R15. Not to bad for two 200$ CPUs

 

However the MoBo has massive coil whine

It's even louder than the two Noctua fans on max speed! Here is a sample if you like: https://www.dropbox.com/sh/srbacaffpmj4b1y/AADeAuNx3Q9HkL9C76jn6miUa?dl=0

The oil can dampen some coil whine, but it will never be aber to silence that amount

 

If you know how to fiy it, please give me a hand here:

 

Thank you.

 

 

Update 2: Design finalized

Spoiler

Due to the change of the heat sinks I had to redo the design work. I decreased the heat sink weigth from ~64kg to ~31 kg, but increased the oil volume from 30.8 liter to 44.8 liter. As I can take the oil out to move the build, I will actually be able to move it once assembeled.

The size is 480x400x400mm.

 

 

 

 

Just for comparison, this was the first version:

 

Update 3: Xeon Phi

Spoiler

I disassemebled the Xeon Phi to prepare the cooling. Somebody at Intel knows a decent amount about cooling and got a huge budget to do his job. Nvidia and AMD should learn from this example!

 

The backplate makes the card very stiff and it makes direct contact with the VRAM chips on the back. Also there are heat pipes embedded in to back plate and there are two copper heat bridges to the main cooler on the front side. Looking at you Titan X with VRAMs on the back in the 90°C range.

 

 

 

The coolir itself is made out of copper and aluminum and has two more heat pipes. And it also makes direct contact to all important components:

 

 

 

And the core itself is cooled by a big vapor chamber with copper finns direct soldered onto it. I assume the 200$ intel sells this card for is just the BOM cost or even less.

 

 

I will try to cut off the finns from the vapor chamber to reuse it. But I only have one singel try and it is very vulnerable.

 

Also I like the blue shroud much more than the aggressive green ore red look of the GPUs from some unnamed manufacturers.

 

 

 

Update 4: Vapor chamber

Spoiler

I cut off the finns of the heat sink of the Xeon Phi. As it has a vapor chamber as a base plate I have now a perfect heat spreader for the Xeon Phi core. And it was even free

 

Update 5: 40 kg heat sinks

Spoiler

The heat sinks have been delivered today. I could barly lift the packet

 

 

 

 

 

Update 6: 73 cores, 450 watts, 846 GFlops

Spoiler

Xeon Phi on Windows 10 is a freaking nightmare! MATLAB somehow don't use the automatic offlod feature despite it is set to do so. At least the linpack benchmark is using it, so and I know the HW is working. The Xeon Phi uses up to 270 watts (@iamdarkyoshi: that's the same as the 390X ) but as I was able to salvage the phase chamber from the original cooler I can spread out the heat enougth to handle it.

And when the reported power draw of the PC is increasing by 250 watts you know it is running. But by burning througth that much power, it manages to score 846 GFlops! This may not look impressive compared to a GPU, but keep in minde this are CPU cores. A i7-6600k has about 200 GFlops and we are talking about >3 year old hardware here.

 

Benchmark results.txt

 

 

 

 

Update 7: How to submerge a HDD

 

 

After a while finaly a new update!

SSDs are really nice for silent systems, but I'm not decadent enougth to store several TBs of cold data on an SSD. So I have to use a HDD. However, running it outside of the oil is not possible as the noise is way to high compared to the rest of the parts. So it needs to be submerged.

I hade a long discussion with Cpatain_WD about if it save to submerge a HelioSeald WD Red. The answer was "in theory it should work, but we don't guarantee it". So I coated the HDD with an electronic protection spray. And I also made a ani-vibration mount with springs to supress the vibration transfer from the HDD to the case.

Springs do have a low dampening so it is endangered to hit the resonant frequency and move heavily. But the surrounding oil will add losses the mechanical system and reduce the reconant peak by a lot.

 

 

 

 

I also drilled most of the necessary holes into the heat sinks, so I can assemble them. This is only for transportation on the 16. June. The sealing is still missing, and I will take the PC apart again once I moved it with the ship across the lake. There will not be a lot of changes for the next 6 weeks as I'm super busy at the moment. Doing funn stuff like spending days at IKEA and looking for used furniture and collecting it takes a lot of time.

At least the electronic is up and running and I'm using it daily.

 

 

 

 

Update 8: It realy hurts doing this!

Spoiler

I found a good deal on a used Titan X so I counl'd resist to buy it. However, after testing I know why it was so cheap: The cooler is very dusty and the card thermal throttles all over the place unless you set the fan to 95%. Stupidly loud, but the GPU itself is working fine. That's all I need.

Sadly Nvidia still used this terrible DVI connector so I had to cut it off. When a hacksaw is your only tool available , it  looks as bad as when Linus is doing hardware modding. You may cried inside when you whatched Linus doing it in the video. But doing it yourself is even worse.

 

 

 

 

Update 9: Moved the build across a lake

Spoiler

So I moved to a new flat that can only be be reached by ship. It is not an island, but sandwiched between the lake and mountains so there is no street. Without the oil it was not even to hard to move the big block of metal that is supposed to be a PC.

 

 

 

 

 

 

 

 

 

 

 

Update 10: Assembly

Spoiler

So I have a lot of work to do and therefor I need my workstation up and runnig. So I finished it as fast as possible.

I'm a bit paranoid about leaking so I used dual or even triple sealing whereover possible. Sadly the oil and temperature resistant sealing was only aviable in brown-red, but I can paint the outside black again later.

 

The system is a ligth load at the moment to speed up the drying process of all the adhesive and sealing used. I will fill it with oil in a couple of hours.

The only thing still missung is the metal stip to protect the IOs of the GPU and the glas plate on top.

 

 

 

 

 

 

 

 

 

 

 

Update 11: Filled the PC with 40 liter of oil!

Spoiler

So I finally filled the pot with medical grade mineral oil and I needed exactly 40 liter.

However, i have bad news: It is leaking. Not very much, I can catch all leaked oil of the last four hours with a single paper towel but still. And I'm not sure how to seal it. The in- and outside already have lots of silicone seal, so adding more won't help much. I', thinking about casting the bottom 3 cm of the build.

The second bad news is power consumption. With Prime95 and Furmark concurently running, the PC pulls about 900 watts from the wall. This is significantly more than the estimated 600 - 650 watts. The stress test is still running, but it won't be able to get rid of next to 1kW passivly.

The reason is the Xeon Phi with an idle power consuption of 120 watts. And Prime95 is pushing the CPU beyond the rated TDP.

 

But for now, enjoy the pictures:

 

 

 

 

 

 

 

Update 12: Finished: Pictures and summary!

Spoiler

 

 

So this week I got the glas cover and finished my build over halfe a year after I started building it. And I have to say I really like the outcome

Have a look by yourself:

 

 

 

 

 

 

 

 

As the build is running oil cooled for almost two month now I can already give you a summary:

 

 

The good....

In the beginning I was not sure if I'm able to master this risky project. Despite a long planing phase, there was a high probabillity ending up with a design flaw and it wouldn't work at all. But the cooling works well, on idle the CPUs are at about 42 C and the GPU at 33C. Now I'm folding at full power on CPU and GPU, both overclocked (CPU 3,1 GHz all cores, GPU 1.277 GHz) and the CPUs are at 64C and the GPU at 73C. I'm only folding since 45 minutes so the temperature will rise a bit more, but that's fine, considering I have the max. possible load and burning over 600 watts.... Also, passive cooled and overclocked Titan X (Maxwell)!

Now I have a quite powerful PC that I don't need to upgrade for several years. And by the way, when I will upgrade I have to dissassembe the entire PC because I can't reach several cables. But I will worry about that in 4 years or so

 

 

... the bad....

There are a few things that didn't turn out like they where intendet. From an aestetic point of view I should have switch the position of the Xeon Phi and the GPU so the DP cables would be on the side and not in front. I assumed there must be a 90 degree DP cable somewhere, but didn't finde any, even in the Shenzhen electronic market, and they have about 3 floors with only display cables. But the PCIe slot allocation is better this way, that's why I choosed this placement in the first way.

Speaking of the Xeon Phi, I'm able to use the auto-offload feature and speed up calculations when using MATLAB, but only when the jobs are big enougth. As I usually have not very high maxtrix / array sizes and low arithmetic desity, hardly any calculation gets send to the Phi. On the other hand the two E5-2680 are already crushing my old machine (i7-4790). Everytime I boot the PC, the Phi will boot as well and stay in the P0 power mode (about 200 watts!) and I have to manually connect to it so it will go dwon to P3. But even then it still burns about 70 watts while waiting for work. This is a lot of energy and I decided to unplug the PCIe cable to turn the card completly off.

When I have a big project in the future I will reconnect it (takes be about 5 minutes + 5 minutes to clean my hands).

Also the adhesive of the cable ties got disolved in the oil and they get lose. The once with screwes are much better, but I liked to drill as less holes througth the heat sinks as possible. By the way, the rubber dampenings on the noctua fans are completly fine.

 

 

... and the ugly.

I used to much thermal paste. Usually this is not a problem as I use nonconductive one. But not the oil dissolved it and it felt down onto the RAM slots.

 

I tried to clean it, but it just dissolved into the oil even more and make it cloudy. I hope it will fall out over time and accumulate on the bottom where it is invisible.

But my worst mistake was to use silicon sealing. As Christoph told me, it is hydrophobical but lipophilic. So it will seal good for water but bad for oil. The oil can go through even the most narrow gap and even diffuses through solid silicone. As a result my build is leaking a little bit. It is not much at all and all gets captured by the catch basin, but still. After one month the oil level in the basin was about 1 mm. I can handle it as I have enough oil, so I will not disassemble everything to fix it. If you work on a similar project keep this detail in mind.

 

 

But in the end I'm really pleased with the outcome.

 

 

[iamdarkyoshi]

So you managed to turn the idea into a reality?

[Stefan1024]

3 minutes ago, iamdarkyoshi said:

So you managed to turn the idea into a reality?

Yes, finally I do

[Stadin6]

u crasy

 

 

 

in a good way

[HelplmChoking]

Followed! Can't wait!

[Nup]

Your build logs get cooler and cooler, looking very much forward to all of this! 

Also; i'm not sure what "like" to give this, theres no badass option or the likes... this needs to be a thing. 

[Stefan1024]

8 hours ago, Nup said:

Your build logs get cooler and cooler, looking very much forward to all of this! 

Also; i'm not sure what "like" to give this, theres no badass option or the likes... this needs to be a thing. 

 

I would say good point is appropriate. But yes, the old like was far easier

[Ddave]

You really love some passive cooling huh ?
Ye rich bastard. LOL

Anyways, your full passive build log got my curiosity, now you got my attention.

Followed !

[Stefan1024]

2 minutes ago, Ddave said:

You really love some passive cooling huh ?
Ye rich bastard. LOL

Anyways, your full passive build log got my curiosity, now you got my attention.

Followed !

Yes I do.

Also I'm selling the old builds, so I do get cash for the new once.

[You_are_a_cunt]

Must see this happening

[Stefan1024]

New renderings:

[Tobleh]

Duuuude I'm so stoked for this build!


Also, those renderings are absolutely beautiful.

[Stefan1024]

2 minutes ago, TobiMay said:

Also, those renderings are absolutely beautiful.

 

Thank you.

But to be fair, I used a lot of models provided by: http://www.grabcad.com

[IMPERIUS]

Crazy. I like it

What will you be using it for?

[Stefan1024]

5 minutes ago, IMPERIUS said:

Crazy. I like it

What will you be using it for?

I had to run some simualtions lately that took over 12 hours each. For situation like this, this number crucher will be handy. Also it will be used a lot for folding at home. And gaming from time to time.

Honestly, I was doing way to much theoretical and software based stuff at work and study over the last several month. So I need a hardware project to work on. It may sound funny, but machining metal is a good work out for me. Also I really wanted to do something with mineral oil for a long time.

As a result, I sold my other rig and sooked for some affordable stuff on ebay. And here I am.

[IMPERIUS]

4 hours ago, Stefan1024 said:

I had to run some simualtions lately that took over 12 hours each. For situation like this, this number crucher will be handy. Also it will be used a lot for folding at home. And gaming from time to time.

Honestly, I was doing way to much theoretical and software based stuff at work and study over the last several month. So I need a hardware project to work on. It may sound funny, but machining metal is a good work out for me. Also I really wanted to do something with mineral oil for a long time.

As a result, I sold my other rig and sooked for some affordable stuff on ebay. And here I am.

Seems cool, just out of curiosity, what do you do for your job? what types of simulations?

[Stefan1024]

Just now, IMPERIUS said:

Seems cool, just out of curiosity, what do you do for your job? what types of simulations?

I'm in the middle of my master degree in electronic engineering. But I only study half time and the other half I'm working for a company that develops electronic equipment for other companies.

There I developt an algorithm that can detect an acoustic alarm even if it's so noisy a human ear can't heat it. As people die when the system fails, we had to do very rigid and excessive testing (simulation).

[Trey222]

This build is going to be awesome and it will be super powerful and have a pretty good price/performance. 

[iamdarkyoshi]

I am excited to see where this build will go! By your calculations, how much surface area on a black anodized aluminum heatsink with heat pipes (if the PC has all fins vertical, for optimal passive airflow) would I need to keep a 3930k and 390x cool at 75 degrees? I want to get two large heatsinks and put the GPU on one and the motherboard on the other. Yes, I have plans on how to do this  I just want to know if it would be at all feasible. I want a nice compact PC, but watercooling is surprisingly bulky.

[Stefan1024]

6 minutes ago, iamdarkyoshi said:

I am excited to see where this build will go! By your calculations, how much surface area on a black anodized aluminum heatsink with heat pipes (if the PC has all fins vertical, for optimal passive airflow) would I need to keep a 3930k and 390x cool at 75 degrees? I want to get two large heatsinks and put the GPU on one and the motherboard on the other. Yes, I have plans on how to do this  I just want to know if it would be at all feasible. I want a nice compact PC, but watercooling is surprisingly bulky.

You are looking at 400-450 watts without OC. You need about two 400mm long chunks of the heat sink i used. This equals to about 3m^2.

 

Also the base plat is so thik it can distrubute the heat over the whole heat sink without heat pipes. but you probably need some to tranport the heat from the CPU / GPU to the heat sink. Also I recommend using a copper plate for the GPU like I did in the Silent cube. Heat pipes are difficult to handle and expensive.

 

[iamdarkyoshi]

1 minute ago, Stefan1024 said:

You are looking at 400-450 watts without OC. You need about two 400mm long chunks of the heat sink i used. This equals to about 3m^2.

 

Also the base plat is so thik it can distrubute the heat over the whole heat sink without heat pipes. but you probably need some to tranport the heat from the CPU / GPU to the heat sink. Also I recommend using a copper plate for the GPU like I did in the Silent cube. Heat pipes are difficult to handle and expensive.

 

I was planning on making a full cover waterblock for the motherboard, except replacing the waterblock with aluminum or copper. Basically put blocks on the hot bits

And on the other side, I would put the GPU on the other heatsink with a pcie riser cable. I need my craigslist PC to sell...

[Stefan1024]

1 minute ago, iamdarkyoshi said:

I was planning on making a full cover waterblock for the motherboard, except replacing the waterblock with aluminum or copper. Basically put blocks on the hot bits

And on the other side, I would put the GPU on the other heatsink with a pcie riser cable. I need my craigslist PC to sell...

I put a lot of effort into finding a way to connect the CPU to the big heat sink without heat pipes. All my solutions would have requred about 10 copper or a very inconvenient placement. Finaly I gave up and just used normal CPU coolers as I will submerge the build in oil anyway. I'm interested to see your solution.

[iamdarkyoshi]

1 minute ago, Stefan1024 said:

I put a lot of effort into finding a way to connect the CPU to the big heat sink without heat pipes. All my solutions would have requred about 10 copper or a very inconvenient placement. Finaly I gave up and just used normal CPU coolers as I will submerge the build in oil anyway. I'm interested to see your solution.

So I just got an idea. I have a bin with AT LEAST 50 pounds worth of aluminum heatsinks. I should totally CAST the heatsinks. Sure, it wouldnt be pretty, but it woukd be interesting and cheap to say the least. Then I could cast adapter plate things to make the hot parts of the motherboard all one flat surface, and find someone with an old clapped out bridgeport to machine flat surfaces  

[Stefan1024]

3 minutes ago, iamdarkyoshi said:

So I just got an idea. I have a bin with AT LEAST 50 pounds worth of aluminum heatsinks. I should totally CAST the heatsinks. Sure, it wouldnt be pretty, but it woukd be interesting and cheap to say the least. Then I could cast adapter plate things to make the hot parts of the motherboard all one flat surface, and find someone with an old clapped out bridgeport to machine flat surfaces  

I'd LOVE to see this build log!

 

EDIT: @iamdarkyoshi 50 pounds is just enougth and not to much

[iamdarkyoshi]

1 minute ago, Stefan1024 said:

I'd LOVE to see this build log!

I will need to think about it. I would need to finish my 3D printer, I would need to get out my aluminum casting stuff, get propane, and plenty of other things. Might have to do it this summer...