Microsoft is submerging servers in boiling liquid

[Lightwreather]

Summary

 

Microsoft has revealed that it's been experimenting with a “two-phase immersion cooling technology” to prevent its data centre servers from overheating and causing outages across its cloud-based communications platforms such as Microsoft Teams. The tech giant is said to be the first cloud provider running two-phase immersion cooling in a production environment.

Quotes

Quote

Microsoft has been testing two-phase liquid immersion cooling technology for a number of years. Now, it's starting to implement this technology in its Azure datacenters, starting with its Quincy, Wash.-based ones, officials said on April 6.

Microsoft has revealed that it's been experimenting with a “two-phase immersion cooling technology” to prevent its data centre servers from overheating and causing outages across its cloud-based communications platforms such as Microsoft Teams. The tech giant is said to be the first cloud provider running two-phase immersion cooling in a production environment.

 

At a Microsoft data centre on the bank of the Columbia River in Washington state, engineers are submerging servers in a steel holding tank filled with boiling liquid.

Unlike water, which is seen as precarious to electronic equipment, the liquid used by Microsoft engineers is harmless to the server hardware as it’s designed to cool its processors by carrying away the heat as it boils.

 

The liquid has been engineered to boil at 50 degrees Celsius, which is 50 degrees cooler than the boiling point of water. The lower temperature has been specifically chosen in order to carry away the heat while allowing Microsoft’s servers to operate at full power without the risk of failure due to overheating.

ccording to Microsoft’s data centre advanced development group VP Christian Belady, “air cooling is not enough”. 

“That’s what’s driving us to immersion cooling, where we can directly boil off the surfaces of the chip,” he said.

 

Husam Alissa, a principal hardware engineer on Microsoft’s team for data centre advanced development, said that the tech giant is “the first cloud provider that is running two-phase immersion cooling in a production environment,” which is the next step in Microsoft’s long-term plan to keep up with the increasing demand for cloud computing.Using specially-designed liquid to cool the servers is not only cheaper than air cooling but also more sustainable than using water, allowing Microsoft to meet its commitment to replenish more water than it consumes by 2030.

 

Belady noted that Microsoft's "Project Natick" undersea data center work demonstrated the improved reliability of systems when humidity and oxygen are removed from an environment. Failures due to corrosion decline substantially.
"With immersion, you have a similar thing. Essentially, you are displacing oxygen and moisture," he said

 

My thoughts

This is actually pretty interesting. Microsoft being the first to implement to implement a rather new technology. Two-phase cooling is actually pretty interesting at least for me but it is unlikely for this tech to trickle down into the consumer spae but I'm sure Linus will find a way to get his hands on this tech. But yea, microsoft becoming sustainable is a good initiative and here's to hoping that other companies follow suit with technology and practices that actually work.

 

Sources

https://www.zdnet.com/article/microsoft-gets-two-phase-immersion-cooling-running-in-an-azure-data-center/

https://www.itpro.co.uk/server-storage/data-centres/359129/microsoft-submerges-servers-in-boiling-liquid-to-prevent-teams

[Ash_Kechummm]

that is some cool tech ngl

 

[Master Disaster]

 

This has been a thing for many many years now. That said its the first time I've heard about it being used on entire enterprise grade servers though I believe it was designed exactly for this use.

 

The stuff they use is called Novec by 3M, you can buy a small bottle for a pretty cheap price.

[Letgomyleghoe]

is this their "fix" to the azure outages, lel.

[Master Disaster]

4 minutes ago, James Evens said:

New? 

submerged is quite a old tech.

phase change is newer but again it is years, maybe decade, old.

 

One problem they have why this won't happen is beside cost regulation on the substances (prohibiting even selling it to consumer).

Cost is not as bad as you think, especially when you consider the savings in electricity over the period of the hardwares life from running cooling.

 

Regulations? Its pretty low toxicity and not all that dangerous. Any good chemical supply company could get hold of a bottle for you, heck you can readily buy things MUCH more dangerous than this stuff from any hardware store.

[Master Disaster]

36 minutes ago, James Evens said:

For a datacenter it is fine. Another large benefit is power density.

Consumer application there are just too many problems: you need to seal the case, pass cables through, etc.

Oh 100% agreed, while its not a particularly dangerous chemical its certainly not something your average Joe/Jill gamer should even consider using at home.

 

I'd love to see Linus get hold of a gallon and test it out with some hardware though, would be a very cool video to watch.

 

I actually wonder if 3M would come over to LMG HQ (obviously after the human malware is under control) and allow LMG to run some tests with it. That way it could be controlled by someone with the required qualifications to handle chemicals in a safe way.

Quote

 

Global warming potential/ozone and with that regulation.

 

Novec is marketed as being Ozone safe, its one the main selling points in the marketing.

 

https://www.3m.co.uk/3M/en_GB/novec-uk/applications/immersion-cooling/

Quote

• Favourable environmental profile
• Low global warming potential (GWP)
• Zero ozone depletion potential (ODP)

 

[Vishera]

6 hours ago, J-from-Nucleon said:

to prevent its data centre servers from overheating

That means that when their servers are pinged to 100% they overheat,that cost cutting ended up costing them more than they saved.

[tkitch]

25 minutes ago, James Evens said:

@Master Disaster Look at the datasheet. The GWP goes upto 500 depending on boiling point .

I am not sure if they are all still highly fluorinated hydrocarbons/ester.

https://forensicssource.com/products/3m-novec-engineered-fluid-hfe-7100-gallon

 

No Ozone Depletion, although at 500$ a gallon?  That's a pretty good speed bump to keep enthusiasts out of the sandbox of this stuff.  

[leadeater]

10 hours ago, James Evens said:

@Master Disaster Look at the datasheet. The GWP goes upto 500 depending on boiling point .

I am not sure if they are all still highly fluorinated hydrocarbons/ester.

You still have to compare that against other cooling methods. If you were to compare against say traditional old refrigerant cooling then it's a landslide win in every category, not that super large deployments use this cooling method anymore. But I'm sure Microsoft has done the calculation against what they are currently doing and it's a net equal or decrease otherwise they wouldn't do it, not unless the current method simply cannot cool the equipment anymore (which I doubt).

 

Plus we know the boiling point, 50C as stated.

[bcredeur97]

MS really has problems with servers overheating causing outages? 

[Arika]

1 minute ago, bcredeur97 said:

MS really has problems with servers overheating causing outages? 

No, they made that up just so they can dip their servers in a random liquid for shits and giggles.

Spoiler

/s

 

Office 365, and therefor sharepoint, goes down every few months, it's really annoying at work, so i do hope this fixes that

[BuckGup]

This is Fluorinert and has been around since the late 1950s. This isn't new or practical as it's expensive and the disposal process is a royal pain in the ass. Everyone just uses liquid cooling now and I highly doubt someone as large as Microsoft is having cooling issues. 

[leadeater]

52 minutes ago, BuckGup said:

This is Fluorinert and has been around since the late 1950s. This isn't new or practical as it's expensive and the disposal process is a royal pain in the ass. Everyone just uses liquid cooling now and I highly doubt someone as large as Microsoft is having cooling issues. 

Though in saying that I doubt it's the exact same as the 1950's or even 10-20 years ago, it'll be something tailored a little to Microsoft requirements. The standard Perfluorohexane boiling point is 56C and what Microsoft are using is supposed to be 50C (unless that's not accurate and it's actually 56C then it is the same )

[BuckGup]

2 minutes ago, leadeater said:

Though in saying that I doubt it's the exact same as the 1950's or even 10-20 years ago, it'll be something tailored a little to Microsoft requirements. The standard Perfluorohexane boiling point is 56C and what Microsoft are using is supposed to be 50C (unless that's not accurate and it's actually 56C then it is the same )

I hope it's not exactly the same lol. I'm more curious why they are having cooling issues and think this is the best solution. 

[leadeater]

5 minutes ago, BuckGup said:

I hope it's not exactly the same lol. I'm more curious why they are having cooling issues and think this is the best solution. 

Maybe this? https://www.3m.com/3M/en_US/company-us/all-3m-products/~/3M-Novec-649-Engineered-Fluid/?N=5002385+3290667401&rt=rud

 

Edit:

All Novec 649:

https://archive.ph/5q8hd

https://indico.cern.ch/event/444246/attachments/1151300/1652860/FI_presentation_8.09.2015.pdf

https://twiki.cern.ch/twiki/pub/LHCb/C6K/Novec_Memo_1.2.pdf

[leadeater]

11 minutes ago, BuckGup said:

I'm more curious why they are having cooling issues and think this is the best solution

I'm sort of on the fence to if they are or not, we switched over to hot aisle containment water heat exchange to outside (not back of rack clip on style) and while it works really well you need very good air flow both inside the hot aisle (getting heat to the water radiators is critical) and then for the outside units good air circulation around those as well. And on top of that you need a lot of surface area so it's not area/space efficient. Then to top that off at each heat exchange phase is about 2C loss so for our 2 phase system the inside cold aisle is 4C hotter than the outside temperature. 

 

If it gets hot water misters turn on and spray the outside units and the evaporation cools them down and help lower the inside temperature. Or emergency there is also straight bypass to run mains water through the system and dump to drain but we've never had to do this.

 

So while you can chase those very nice PUE numbers there can actually be some real trade offs, like we have. Being in New Zealand and not having very hot summers non chilled water cooling works effectively, anywhere else then not at all. Chilled water still uses a lot of energy so you aren't going to get any PUE awards doing that.

 

Basically right now in the industry there is a little too much chasing the PUE figures and not enough practical application consideration i.e. PUE 1.03 but your servers might have heat issues and you need to load shift vs PUE 1.05 with no caveats I would argue the 1.05 design is better (very broadly lol).

[GDRRiley]

4 minutes ago, leadeater said:

So while you can chase those very nice PUE numbers there can actually be some real trade offs, like we have. Being in New Zealand and not having very hot summers non chilled water cooling works effectively, anywhere else then not at all. Chilled water still uses a lot of energy so you aren't going to get any PUE awards doing that.

 

Basically right now in the industry there is a little too much chasing the PUE figures and not enough practical application consideration i.e. PUE 1.03 but your servers might have heat issues and you need to load shift vs PUE 1.05 with no caveats I would argue the 1.05 design is better (very broadly lol).

last I looked we were at 1.07 with just hot and cold isles but HPC section has rads between servers and our new system is direct water cooling.

 

I'd also remind you that PUE for submerged cooling isn't the same as normal PUE due to the lack of fans

[BuckGup]

15 minutes ago, leadeater said:

I'm sort of on the fence to if they are or not, we switched over to hot aisle containment water heat exchange to outside (not back of rack clip on style) and while it works really well you need very good air flow both inside the hot aisle (getting heat to the water radiators is critical) and then for the outside units good air circulation around those as well. And on top of that you need a lot of surface area so it's not area/space efficient. Then to top that off at each heat exchange phase is about 2C loss so for our 2 phase system the inside cold aisle is 4C hotter than the outside temperature. 

 

If it gets hot water misters turn on and spray the outside units and the evaporation cools them down and help lower the inside temperature. Or emergency there is also straight bypass to run mains water through the system and dump to drain but we've never had to do this.

 

So while you can chase those very nice PUE numbers there can actually be some real trade offs, like we have. Being in New Zealand and not having very hot summers non chilled water cooling works effectively, anywhere else then not at all. Chilled water still uses a lot of energy so you aren't going to get any PUE awards doing that.

 

Basically right now in the industry there is a little too much chasing the PUE figures and not enough practical application consideration i.e. PUE 1.03 but your servers might have heat issues and you need to load shift vs PUE 1.05 with no caveats I would argue the 1.05 design is better (very broadly lol).

Interesting, so by hot and cold aisle do you mean blowing cold air from the floor vents in front of the racks and then the aisle behind is considered hot? Also I'm curious how the misters work. At HPE the go to now is watercool everything in a hot swappable blade like Badger or Mountain and have very high flow rates and expel all the heat outside in a massive cooling unit. HPE just upgraded their facility in wisconsin and now are the second biggest power consumers in the state. 

[tkitch]

3 hours ago, BuckGup said:

Interesting, so by hot and cold aisle do you mean blowing cold air from the floor vents in front of the racks and then the aisle behind is considered hot? Also I'm curious how the misters work. At HPE the go to now is watercool everything in a hot swappable blade like Badger or Mountain and have very high flow rates and expel all the heat outside in a massive cooling unit. HPE just upgraded their facility in wisconsin and now are the second biggest power consumers in the state. 

in a GOOD datacenter (not all do this) - a COLD aisle is the air intake side.  (Both rack rows there should intake from the same side, generally the "front" of the rack.)

Hot is the Exhaust side.  (Which should be both sides of a given aisle) 

[Distinctly Average]

6 hours ago, leadeater said:

You still have to compare that against other cooling methods. If you were to compare against say traditional old refrigerant cooling then it's a landslide win in every category, not that super large deployments use this cooling method anymore. But I'm sure Microsoft has done the calculation against what they are currently doing and it's a net equal or degrease otherwise they wouldn't do it, not unless the current method simply cannot cool the equipment anymore (which I doubt).

 

Plus we know the boiling point, 50C as stated.

Other cooling methods are currently being experimented with as well as combinations of them. Slapping them in the sea seems to be quite popular. There are even plans to experiment with orbiting data centres but that sounds extremely expensive as well as latency bound. Plenty of liquids have been tried and tested too. However, maybe there needs to be a technology change to reduce heat production in the first place.

[StDragon]

I would love to have this in a sealed PC case. Because of uniform thermal management due to mass, it would be easier to control hots spots as well as cool the outside of the case (the other side) with no dust intrusion.

[Taf the Ghost]

Thoughts on the "why?": energy costs for cooling are deeply affected by the local environment. As the industry has moved to put large datacenters everywhere, it makes sense to design special case solutions for places with much harsh environments or energy costs. Though it strikes me this is more about Power than Cost, as local government approval for datacenters can easily run into issues around power draw. If you can offload the cost of Cooling from Electricity costs into the costs of the Liquid at a high enough rate, it changes the economics (and potentially politics) around a datacenter. 

 

I'm also sure it's a study for how to cram more servers into a smaller area. All about that Compute Density.

[leadeater]

7 hours ago, GDRRiley said:

I'd also remind you that PUE for submerged cooling isn't the same as normal PUE due to the lack of fans

Well it is the same, just lower. With submerged cooling you can chase PUE figures below 1.03, but it's a lot of hassle tbh. PUE is just ratio of equipment power to facility power so while you do lower the equipment power by not having fans in them it doesn't change how you calculate PUE.

[leadeater]

7 hours ago, BuckGup said:

Interesting, so by hot and cold aisle do you mean blowing cold air from the floor vents in front of the racks and then the aisle behind is considered hot?

No, that's well the more old school way of doing it and those vents would be cold air fed by AC units, how we were doing it. Hot aisle containment means that you actually seal off the hot aisle and the only path for the air to move through is through a large stack of radiators. So pretty much the entire wall on one side of our DC is rads and fans that blows the air that was hot back in to the room as the room itself is the "cold aisle".

 

The equipment fans themselves largely do the work of drawing the air through the equipment in to the hot aisle which creates a high pressure zone within that and a low pressure zone in the cold aisle. We have pressure sensors through the hot and cold aisle and these along with temperature probes control the fan speeds on the inside rads and the outside rads.

 

So for us it's really important to close the doors when we go in to the hot aisle or the whole thing pretty much stops working, they self close 

 

Edit:

I'll get you some pictures next week. It's actually a really basic design, ceiling space is part of the hot aisle btw, that's how the hot air travels to get to the rads/fans wall area to be blown back in to the room. The heat is transferred in to the water as it goes through those rads and then the water is circulated to rads/fans on the roof.

[leadeater]

7 hours ago, GDRRiley said:

last I looked we were at 1.07 with just hot and cold isles but HPC section has rads between servers and our new system is direct water cooling.

Our room is too small for in row water cooling with no chilling, some of smaller network distributions areas are cooled this way but are in row AC as it would actually be a real pain and expensive to build containment zones in to the actual hundreds (possible very low thousand) network distribution areas.

 

So we have to heat exchange to the outside. We don't actually know what our PUE number is and I don't think we can be bothered to calculate it lol. But we have gone from AC cooling to just water pumps and fans and reduced the total facility power load by like 50% due to that, which is nice for other reasons as the AC units were on UPS power and generator so we gained back a huge amount of capacity there.

 

So having a low PUE can be very beneficial I just think there should be caution in trying to get it too low, at some point you'll have to make compromises to be able to get it lower and some of those compromises aren't very worker friendly or require a lower power density per rack.

 

P.S By anywhere else I mean anywhere that's always hot AF or can get hot AF. So basically any place that doesn't get hotter than 36C as that would mean 40C inside room temperature which is a rather typical upper maximum inlet temperature for equipment.