Microsoft is submerging servers in boiling liquid

[wasab]

1 hour ago, Taf the Ghost said:

Thoughts on the "why?": energy costs for cooling are deeply affected by the local environment. As the industry has moved to put large datacenters everywhere, it makes sense to design special case solutions for places with much harsh environments or energy costs. 

Indeed. Am I the only one with the brilliant idea of building a data center in Antarctica? You need zero active cooling. 

[GDRRiley]

34 minutes ago, leadeater said:

Our room is too small for in row water cooling with no chilling, some of smaller network distributions areas are cooled this way but are in row AC as it would actually be a real pain and expensive to build containment zones in to the actual hundreds (possible very low thousand) network distribution areas.

 

So we have to heat exchange to the outside. We don't actually know what our PUE number is and I don't think we can be bothered to calculate it lol. But we have gone from AC cooling to just water pumps and fans and reduced the total facility power load by like 50% due to that, which is nice for other reasons as the AC units were on UPS power and generator so we gained back a huge amount of capacity there.

 

So having a low PUE can be very beneficial I just think there should be caution in trying to get it too low, at some point you'll have to make compromises to be able to get it lower and some of those compromises aren't very worker friendly or require a lower power density per rack.

 

 

yeah I know you've got a tiny DC.

 

I'm really hoping after we get our current system in we can do something like the LTT cedar supercomputer video internally, which would be super cool.

 

We've got a few guys whos only job is to lower our power and water usage by monitoring and the back end for stuff like grafana. why the hell was AC on UPS that doesn't make sense. fans I would get but AC too wtf?

 

realize that my lab for the last 15 years has been pushing the PUE as low as we can. we keep pushing our density up and I wouldn't say anything we do is a pain. our new system is 300-400KWA per rack

 

1 minute ago, wasab said:

Indeed. Am I the only one with the brilliant idea of building a data center in Antarctica? You need zero cooling. 

you've got issues that come with that like struggle to get power, expensive to ship lots of heavy stuff there, and lots of latency. you'd also likely need to heat the DC if you can't keep the power used over a certain point

[leadeater]

3 minutes ago, GDRRiley said:

why the hell was AC on UPS that doesn't make sense. fans I would get but AC too wtf?

They weren't originally but during the DC upgrade we moved the entire building on to the UPS, and generator. Before that AC was on generator backup supply only. But since the building is also our office and we were going to half the power usage we wanted to just put everything on central protected power and we also removed all automatic change over switches as those are the most likely failure points as the can weld themselves when the do a changeover and you don't know they have until they won't change back.

 

All the electrical work had to be done first before anything else could so AC was on UPS for maybe a year.

[GDRRiley]

Just now, leadeater said:

They weren't originally but during the DC upgrade we moved the entire building on to the UPS, and generator. Before that AC was on generator backup supply only. But since the building is also our office and we were going to half the power usage we wanted to just put everything on central protected power and we also removed all automatic change over switches as those are the most likely failure points as the can weld themselves when the do a changeover and you don't know they have until they won't change back.

 

ah okay I was going to say weird.

[leadeater]

13 minutes ago, GDRRiley said:

I wouldn't say anything we do is a pain

But you also aren't trying literally everything at the expense of sanity to try and achieve 1.0 

[GDRRiley]

2 minutes ago, leadeater said:

But you also aren't trying literally everything at the expense of sanity to try and achieve 1.0 

no we've accepted that 1.05-1.10 is good enough

[leadeater]

9 minutes ago, GDRRiley said:

no we've accepted that 1.05-1.10 is good enough

3M's whole pitch about immersion cooling is PUE of 1.02 (with potential to go lower) but I'd certainly say doing that has some tradeoffs. I think the idea is really cool but I wouldn't actually want to have to work with that.

 

Also I really don't mind, personally actually like, that our hot aisle is now 50C+ at all times but other members of my team really do not like it at all lol. One of them was in there for 2 hours, by mistake (lost track of time), and had to take the next day off. I like warmth, looking forward to being able to jump in there in winter

[Taf the Ghost]

3 hours ago, wasab said:

Indeed. Am I the only one with the brilliant idea of building a data center in Antarctica? You need zero active cooling. 

Unless you're doing laser communications, keeping the cables from degrading would be a big worry for the communication layer.

 

That said, I'm really surprised we haven't seen heavy use of vertical heat-pump systems. Microfsoft already has done the in-ocean testing stuff.

[trag1c]

21 hours ago, James Evens said:

One problem they have why this won't com to the consumer space is beside cost regulation on the substances (prohibiting even selling it to consumer).

idk. There's a lot of companies willing to deliver a 55 gallon drum of god knows what to residential addresses. I have a big ole drum of flammable brake parts cleaner sitting in my garage right now. That shit is as toxic and carcinogenic as it gets and I get it delivered to my house no questions asked lol. Same goes for my Oxy Acetylene torch. Acetylene is stupidly volatile on its own never mind having a tank of pure oxygen next to it. A lot of things that you think would be regulated or prohibited are not, you just have to muster up the courage to call your local vendor/supplier. 

 

 

With that said I would imagine a low toxicity coolant would be pretty easy to acquire. 

[Brooksie359]

15 hours ago, tkitch said:

https://forensicssource.com/products/3m-novec-engineered-fluid-hfe-7100-gallon

 

No Ozone Depletion, although at 500$ a gallon?  That's a pretty good speed bump to keep enthusiasts out of the sandbox of this stuff.  

To be fair enthusiasts spends thousands on custom water cooling so its not like there isn't anyone who would do it. Having a custom loop typically makes no practical sense in terms of price vs performance and is more oriented toward people who simply like the building aspect as well as the end product look and feel. I wouldn't doubt there would be some who would want to do something like this. It sorta brings to mind the mineral oil pc build. 

[Taf the Ghost]

42 minutes ago, Brooksie359 said:

To be fair enthusiasts spends thousands on custom water cooling so its not like there isn't anyone who would do it. Having a custom loop typically makes no practical sense in terms of price vs performance and is more oriented toward people who simply like the building aspect as well as the end product look and feel. I wouldn't doubt there would be some who would want to do something like this. It sorta brings to mind the mineral oil pc build. 

We've already seen Novec fluid systems at trade shows. The issue is generally how you're going to design the CPU and/or GPU block. You really don't need to actually submerge the whole system.

[leadeater]

1 hour ago, Brooksie359 said:

I wouldn't doubt there would be some who would want to do something like this.

I mean I do work at a university, known for sciences, with a staff university email address 

 

Just get it delivered to the ITS dockway.. hmmm

[Jet_ski]

Several companies including Microsoft are working on ARM chips for data centers. If the M1 has proven anything, it’s that high performance per watt can be achieved without too high of a thermal output.

 

So why is Microsoft taking this approach instead of implementing the more efficient technologies out there?! This is pretty backwards.

[leadeater]

19 minutes ago, Jet_ski said:

Several companies including Microsoft are working on ARM chips for data centers. If the M1 has proven anything, it’s that high performance per watt can be achieved without too high of a thermal output.

 

So why is Microsoft taking this approach instead of implementing the more efficient technologies out there?! This is pretty backwards.

Because power efficiency and power usage are different things. These companies want high performance, high density which means high power regardless of ISA used.

 

https://www.anandtech.com/show/15871/amperes-product-list-80-cores-up-to-33-ghz-at-250-w-128-core-in-q4

 

ARM server CPUs use just as much power as Intel and AMD x86 CPUs do, and whether it be ARM or x86 they can use less but that's not actually what is wanted.

 

And for all those cores and all that power, granted lower than 7763, it's not faster in FP performance

[PeachGr]

The boiling at 50°C, means that the temperature of that liquid, can't go above that.

Luke had a submerged PC on mineral oil.

I just need an aquarium like that. A motherboard on the bottom and hard drives floating, and an RGB strip somewhere

[HerrKaLeu]

I had to look up PUE. It states top performers get a around 1.2. I assume that includes the need for a heat pump cycle and not just " free"  cooling (economizer) assuming the climate requires heat pump cooling. 

 

How does one get the 1.03 mentioned in this thread? this can only be possible if you are in a cold climate and only need to operate pumps and fans. A good chiller has a COP of maybe 7. So you need about 0.14 for the chiller alone plus auxiliary fans and pumps.  COP will vary with the lift required, which will depend on ambient temperature. 

 

I assume PUE is an annual number. So if you are in a climate with cold winters, you can free-cool part of the year. But this gets really wonky to compare data centers. the Florida Datacenter will inherently have higher numbers than an Alaska center. 

 

The claim to get PUE of 1.03 with this phase change seems to ignore that once the server heats up the liguid (boils it) it needs to be cooled to condense and return to the server. Again, if you are not in a cold climate, you need some heatpump (like a chiller) to dissipate the heat. 

 

From a thermodynamic point this isn't so much different than a heat pipe cooler. Except the entire server is in the heatpipe. and the condensing end still needs to be cooled by "something". the advantage i see due to phase change you need to move less fluid per kW cooled. and due to submersion you also cool all components (capacitors, etc.). So if you need to plug in a new cable you drain the server or send in a scuba diver?

 

Or is this a one-time use of the liquid? You wouldn't need a heat pump if you constantly inject new liquid and let it evaporate. I can't believe this is the case wit the cost of the liquid, and the hazard. 

 

So are they missing part of the energy cost in their claim? Or did they re-write laws of thermodynamics? Or is that just in a cold climate? 

 

[staticpage]

I used first gen Novec to cool a pc many years, maybe decades ago. It was really neat until I found that the Novec was evaporating and condensing in strange places in my home  like window-sill. MSDA looked sorta scary when I realized I was breathing it.

[leadeater]

8 hours ago, HerrKaLeu said:

I had to look up PUE. It states top performers get a around 1.2. I assume that includes the need for a heat pump cycle and not just " free"  cooling (economizer) assuming the climate requires heat pump cooling. 

 

That's just an old figure, there's tons of places below 1.2 now days.

 

Quote

In October 2008, Google's Data center was noted to have a ratio of 1.21 PUE across all 6 of its centers, which at the time was considered as close to perfect as possible. Right behind Google, was Microsoft, which had another notable PUE ratio of 1.22 [13]

 

Since 2015, Switch, the developer of SUPERNAP data centers, has had a third-party audited colocation PUE of 1.18 for its SUPERNAP 7 Las Vegas Nevada facility, with an average cold aisle temp of 20.6C (69F) and average humidity of 40.3%. This is attributed to Switch’s patented hot aisle containment and HVAC technologies.[14]

 

As of the end of Q2 2015, Facebook's Prineville data center had a power usage effectiveness (PUE) of 1.078 and its Forest City data center had a PUE of 1.082.[15]

 

In October 2015, Allied Control has a claimed PUE ratio of 1.02 [16] through the use of two-phase immersion cooling using 3M Novec 7100 fluid.

 

In January 2016, the Green IT Cube in Darmstadt was dedicated with a 1.07 PUE.[17] It uses cold water cooling through the rack doors.

 

In February 2017, Supermicro has announced deployment of its disaggregated MicroBlade systems. An unnamed Fortune 100 company has deployed over 30,000 Supermicro MicroBlade servers at its Silicon Valley data center with a Power Use Effectiveness (PUE) of 1.06.[18]

 

Through proprietary innovations in liquid cooling systems, French hosting company OVH has managed to attain a PUE ratio of 1.09 in its data centers in Europe and North America.[19]

 

 

I would have to assume ours is below 1.2 also since the only cooling power usage is 3 water pumps and 6 (large outside) + 9 (largish inside fans) fans. One of the most reliable methods to lower the PUE is to run a higher cold aisle temperature, 20C is a much older target where as many use 25C-27C today.

[leadeater]

8 hours ago, HerrKaLeu said:

The claim to get PUE of 1.03 with this phase change seems to ignore that once the server heats up the liguid (boils it) it needs to be cooled to condense and return to the server. Again, if you are not in a cold climate, you need some heatpump (like a chiller) to dissipate the heat. 

It doesn't as the boiling point is 50C and there is condensing coils above the liquid in the sealed units and the cooling coil will be cooled most likely with water that is recirculated outside, so as long as it's less than 50C outside the cooling fluid will be contacting a cooling coil that is below 50C and thus condense and drip back down.

 

https://www.3m.com/3M/en_US/data-center-us/applications/immersion-cooling/

[Bombastinator]

On 4/9/2021 at 2:37 AM, Master Disaster said:

Cost is not as bad as you think, especially when you consider the savings in electricity over the period of the hardwares life from running cooling.

 

Regulations? Its pretty low toxicity and not all that dangerous. Any good chemical supply company could get hold of a bottle for you, heck you can readily buy things MUCH more dangerous than this stuff from any hardware store.

My memory is the stuff was created as a fire extinguishing liquid and banned in the EU on environmental grounds.  Or something like that.  I’m remembering another thread about some server farm that burned down in the EU somewhere.  Thing involved buildings made up to look like shipping containers or something.

[leadeater]

4 minutes ago, Bombastinator said:

My memory is the stuff was created as a fire extinguishing liquid and banned in the EU on environmental grounds.  Or something like that.  I’m remembering another thread about some server farm that burned down in the EU somewhere.  Thing involved buildings made up to look like shipping containers or something.

There's many different Novec products, some with and many without environmental harm. Novec 7100, yes not so great where as Novec 649, no problem at all. There isn't a "single Novec".

[Bombastinator]

Seems like potentially a lot of waste heat to get rid of.  Waste of all sorts can often be turned into something valuable.  Just have to package it in a way that it’s good for something.  Before plastic the oil industry produced mountains of useless sludge.  Plastic got invented though and it wasn’t waste anymore.  It was another product.

Edited April 11, 2021 by Bombastinator

[CarlBar]

I suspect it's the same thinking behind Intel's attempts to create a plug and play chiller cooler. Where getting mighty close to running into the physics limits on silicon density. Where still some way off the absolute peak, (and w'e'll be able to optimise beyond that like intel did with 14nm), but where getting very close to the point where any alternative to replace silicon would have to start setting up fabs if it wants to be ready before we hit the wall. Intel, Microsoft, e.t.c. are probably hedging their bets that a replacement won;t be ready in time and we'll have to go to more power and thus heat dense systems with bigger chips. How well are current racks going to cope when you start cramming 4 or more times the power density in them. I imagine not very well.

[Bombastinator]

14 minutes ago, CarlBar said:

I suspect it's the same thinking behind Intel's attempts to create a plug and play chiller cooler. Where getting mighty close to running into the physics limits on silicon density. Where still some way off the absolute peak, (and w'e'll be able to optimise beyond that like intel did with 14nm), but where getting very close to the point where any alternative to replace silicon would have to start setting up fabs if it wants to be ready before we hit the wall. Intel, Microsoft, e.t.c. are probably hedging their bets that a replacement won;t be ready in time and we'll have to go to more power and thus heat dense systems with bigger chips. How well are current racks going to cope when you start cramming 4 or more times the power density in them. I imagine not very well.

This makes sense to me.  There have been a few false starts on that one where people thought silicon was ready to be outdated but new things got found so it wasn’t.  The latest I know of was SOI.  alternative stuff is pretty well researched. There’s a pretty good chance that silicon will be harder to beat than expected.

Edited April 12, 2021 by Bombastinator

[Bombastinator]

2 hours ago, ThatOneDunce said:

They learned from OVH... your servers can't burn if they're all submerged in fluid!

*Points a terrified shaking finger at certain extremely scary fluorine compounds.*