OVH Cloud suffers major fire in Strasbourg datacenter

[leadeater]

44 minutes ago, Bombastinator said:

My understanding of data center containers is they’re not work spaces.

Yep there is a small entry way with very basic storage but not work space and there is only enough space in the main cabinet row to rack equipment and nothing more. Ours really is a shipping container though (reinforced and modified etc) and can be stacked using the shipping container locks (this is why they are popular). Horrible place to be in for any long length of time.

[rand_althor_cometh]

I seriously don't get how an entire datacenter building burns down without containment. Did they not even have a sprinkler system for the containment areas?

[Bombastinator]

1 hour ago, rand_althor_cometh said:

I seriously don't get how an entire datacenter building burns down without containment. Did they not even have a sprinkler system for the containment areas?

Sprinklers  don’t work for electrical fire.  This is why someone was mentioning halon systems.  Halon can kill people though (it’s not air so nothing can burn, but no air isn’t good for humans) so the area has to also be at least nearly airtight to protect the public.  Really hardcore archival ancient book libraries can have them too.  In those situations the book stacks are separately encased.  It’s a really expensive system to put in place and it’s complicated so it has to be tested regularly.  Only one of three buildings burned, but the one that did burn they apparently couldn’t do much with.  Maybe the fire suppression system malfunctioned or it was never in place.  AFAIK the only thing worse is a rocket fuel type fore where the burning matter also produces its own oxygen. I’ve heard them called “road flare” fires.  At least some Lithium batteries can pull that one. All one can do is bury them and wait for them to use up all their fuel and go out. 

Edited March 12, 2021 by Bombastinator

[Bombastinator]

10 minutes ago, James Evens said:

Halon is a FCKW so newer system will use different gases. Areas which have these system might have warning labels when you enter and a acoustic and visual warning with some delay before they start.

I often have to look acronyms up.  Looking that one up was kind of amusing.  It’s apparently a German acronym for hydrocloroflourocarbon, which would be an HCFC in English, and a Pittsburg swim team Called the fox chapel killer whales. I’m assuming the first one. 

[GDRRiley]

On 3/10/2021 at 9:17 PM, Elvara said:

That fire is pretty impressive and that building is pretty recent 2017 I believe,

2011 is when the 2nd was built

[GDRRiley]

there is talk of it maybe being from the UPS, they just had one serviced that day or the day before
https://www.ovh.com/fr/images/sbg/index-en.html

Edited March 12, 2021 by GDRRiley
fixed link

[CarlBar]

2 hours ago, Bombastinator said:

AFAIK the only thing worse is a rocket fuel type fore where the burning matter also produces its own oxygen.

 

Thats less about it being self oxidising as it is the oxidant density. Air has around 0.15 grams of oxygen per litre. Liquid oxygen runs to around 1141 grams per litre, thats around 7000 times as much. There's very little, (including some normally fireproof materials), that won't burn in the presence of that much oxygen. There are stronger oxidisers too, though they've only been investigated experimentally for rocket use, fortunately. Since they'll start fires with all sorts of things, like bricks, sand, asbestos, ashes.

[Bombastinator]

2 hours ago, CarlBar said:

 

Thats less about it being self oxidising as it is the oxidant density. Air has around 0.15 grams of oxygen per litre. Liquid oxygen runs to around 1141 grams per litre, thats around 7000 times as much. There's very little, (including some normally fireproof materials), that won't burn in the presence of that much oxygen. There are stronger oxidisers too, though they've only been investigated experimentally for rocket use, fortunately. Since they'll start fires with all sorts of things, like bricks, sand, asbestos, ashes.

I’m recalling this stuff the Germans used in ww2 that flat out terrified them it was used in this rather strangely bulbous rocket plane that looked a lot like a flying lime.  That stuff might qualify as one of your more energetic oxidizers.   I dunno.  I’m scared enough of things like road flares and thermite.  You can’t get deader than dead.

[Bombastinator]

14 hours ago, Elvara said:

Well the datacenter in container is apparently nothing new and it's used from what I saw on google well even google works or have worked with it https://www.youtube.com/watch?v=zRwPSFpLX8I but it's not the building that caught on fire.

But yeah I didn't think of it like that the building maximizing airflow, would also maximize fire propagation and smoke. I never witness computers starting a fire, there's quite a lot of metal, that's why I was surprised at the size of the fire.
I work in the hotel industry and our fire safety is pretty hardcore from fire detection to what kind of paint or fabric we can use in rooms or corridors also it's mostly fire retardant materials. I know almost everything still burn at some point but I'm surprised how industrial building burn that big specially were they don't stock flammable materials and the interior is most of the time bare walls.

I recall the “storage containers” being on their side. A position in which they don’t have strength so the building might have a storage container facia, but it couldn’t have even resembled one structurally on the inside.

[CarlBar]

12 minutes ago, Bombastinator said:

I’m recalling this stuff the Germans used in ww2 that flat out terrified them it was used in this rather strangely bulbous rocket plane that looked a lot like a flying lime.  That stuff might qualify as one of your more energetic oxidizers.   I dunno.  I’m scared enough of things like road flares and thermite.  You can’t get deader than dead.

 

Thats the one that will set fire to bricks, sand, asbestos, ashes.

 

https://blogs.sciencemag.org/pipeline/archives/2008/02/26/sand_wont_save_you_this_time

 

And yes the Germans experimented with the stuff. Initially for use in self igniting flamethrowers. They found it too nasty to work with for that. Bear in mind they made Sarin nerve gas in industrial quantities.

[leadeater]

4 hours ago, GDRRiley said:

there is talk of it maybe being from the UPS, they just had one serviced that day or the day before
https://www.ovh.com/fr/images/sbg/index-en.html

Huge out of control fires like this is very likely a battery fire. When you've got actually a few tons of batteries and there is a fault that causes a fire (internal battery short or something else) they can catch fire, once the battery starts burning it's very hard to stop it then the rest will eventually go up and then you'll not be able to realistically stop it.

 

I know of multiple cases of battery fires in datacenters, all of them caused by cheaping out on the battery replacement and either using refurbished batteries (yes that is a thing!) or not using approved/certified replacement batteries.

 

For residential systems you have to, at least here, put the batteries in a weather protected enclosure and then have the correct warning labels on it and also notify the fire department that you have one so they know you have large amount of Lead Acid batteries or Lithium Ion batteries.

 

The talk about the power cables just isn't likely, a cable fault like that wouldn't lead to this.

[Bombastinator]

35 minutes ago, leadeater said:

Huge out of control fires like this is very likely a battery fire. When you've got actually a few tons of batteries and there is a fault that causes a fire (internal battery short or something else) they can catch fire, once the battery starts burning it's very hard to stop it then the rest will eventually go up and then you'll not be able to realistically stop it.

 

I know of multiple cases of battery fires in datacenters, all of them caused by cheaping out on the battery replacement and either using refurbished batteries (yes that is a thing!) or not using approved/certified replacement batteries.

 

For residential systems you have to, at least here, put the batteries in a weather protected enclosure and then have the correct warning labels on it and also notify the fire department that you have one so they know you have large amount of Lead Acid batteries or Lithium Ion batteries.

 

The talk about the power cables just isn't likely, a cable fault like that wouldn't lead to this.

Question: in what instances did said fire take down an entire building when it went up?  Trying for a rough estimate of how common it is for a whole building to go when a battery fire starts

[Bombastinator]

1 hour ago, CarlBar said:

 

Thats the one that will set fire to bricks, sand, asbestos, ashes.

 

https://blogs.sciencemag.org/pipeline/archives/2008/02/26/sand_wont_save_you_this_time

 

And yes the Germans experimented with the stuff. Initially for use in self igniting flamethrowers. They found it too nasty to work with for that. Bear in mind they made Sarin nerve gas in industrial quantities.

I heard about it in a blog with the subtitle “things I won’t work with” or some such.  It was a while ago. Memory is vague.  I mostly fixated on the bulbous aero planes.  My brain decides what it will remember and what it won’t.  I have little control

[leadeater]

6 minutes ago, Bombastinator said:

Question: in what instances did said fire take down an entire building when it went up?  Trying for a rough estimate of how common it is for a whole building to go when a battery fire starts

Usually it's not the entire building but those are a bit different to this as the buildings weren't just a datacenter hall but a building with a datacenter facility within it, so much harder for the fire to break out as they have firewalls between the internal building zone types.

 

Our batteries are located in a concrete basement area, the assumption is if they go up in flames extinguishing is impossible, so the idea is to contain and let burn itself out. Smoke from this wouldn't easily be able to make it's way in to the actual server and switch rooms.

 

Our building is also the office area for ITS and has like 100+ people in it so the fire rules are quite strong due to the presence of people.

 

The ones I know about most of the damage was done by the toxic smoke, everything near the batteries were destroyed by heat but the smoke is particularly bad.

 

When we were looking at co-location options we were only looking at the premium hosting providers that had basically buildings within buildings, each data hall was a fire zone with isolation between each so fire wouldn't realistically burn from one to another even if very large, here we also look for earthquake isolation but that's a local thing (ring of fire lol). Anyway OVH is not one of these, they might have some facilities to that standard but the ones in this story are not. I say this without full knowledge of their actual design but the fact that all of SBG2 got destroyed and 4 halls of SBG1 tells me they were not of that standard, or whatever they did failed so therefore were not, and it was also an even bigger failure that fire was able to spread from one building to another.

[CarlBar]

6 hours ago, Bombastinator said:

I heard about it in a blog with the subtitle “things I won’t work with” or some such.  It was a while ago. Memory is vague.  I mostly fixated on the bulbous aero planes.  My brain decides what it will remember and what it won’t.  I have little control

 

Same blog as what i linked.

[Bombastinator]

6 hours ago, CarlBar said:

 

Same blog as what i linked.

So it is.  I didn’t look at the link till you mentioned it. Could be a different chemical I think though.  I googled it and got a different page ( https://www.vedantu.com/chemistry/chlorine-trifluoride ) there was mention in that it was never actually used in war but this stuff actually got used though it was at the very end.  Timing would be right though so I don’t know.  Something first discovered  in Germany in the 30’s. There are statements that it is used as a component in rocket fuel but I didn’t see any examples.  I recognise the phrase “good pair of running shoes” as being mashed together with it though.  I wonder if I’m conflating stuff in my head. Sometimes I hate my brain and it’s little fallibilities.  The article where I first heard of it was talking about nazi experimental aircraft.  There were black and white photos of the plane with pilots sitting in them and discussion of how they were used at the end of the war to attack allied bombing raids. 

 Chlorine triflouride is apparently outlawed by the Geneva convention.  I’m a bit horrified that such a material could also be used in industry or that it is even manufactured.  The film liner creating effective containment thing rings a bell, but there are several things that do that though. This is part of the problem of talking about rarified chemical stuff without actually being a chemist.  There is apparently even a chlorine pentaflouride. Also described as a rocket fuel component. Might be what I originally read about. 

Edited March 13, 2021 by Bombastinator

[CarlBar]

The aircraft your thinking of is the Me-163, i believe ti was CLF3 they used as oxidant, but they where pretty desperate by the end and it was insanely dangerous. AFAIK it's not outright banned in war, it just there's no good justification for using it ATM so it would fall under the needlessly inhumane clause of the conventions.

[Bombastinator]

13 minutes ago, James Evens said:

ClF3 is interesting.

Hydrazine isn't nice but ClF3 is on a entirely different scale.  It violently reacts with a lot of things, need special steel vessels meaning this wouldn't be the first choice if there is a cost effective alternative.

 

next up on your surprise list: phosgene

That name rings a bell as a “stay the hell away from” kinda thing.  “Phosgene gas” was One of those discussion ending terms. *goes to look up phosgene*

 

In the Wikipedia it says phosgene can be created by a combination of carbon monoxide and chlorine gas exposed to sunlight.  So driving an old fashioned car near a swimming pool?  It’s apparently been in use in industry since the 19th century. 

Edited March 13, 2021 by Bombastinator

[CarlBar]

5 hours ago, James Evens said:

ClF3 is interesting.

Hydrazine isn't nice but ClF3 is on a entirely different scale.  It violently reacts with a lot of things, need special steel vessels meaning this wouldn't be the first choice if there is a cost effective alternative.

 

next up on your surprise list: phosgene

 

Funny fact, i visited a chem company in the north of the UK during a collage course, (economics stupidly enough, long drive, ughh, bad memories), and i asked what the nastiest thing they used there was and what it was used for. Answer: Mustard gas. Used at the time as a precursor in the production of Viagra. 

 

Also it's less special steels as it is they need a fluoride oxide layer AFAIK.

 

5 hours ago, Bombastinator said:

That name rings a bell as a “stay the hell away from” kinda thing.  “Phosgene gas” was One of those discussion ending terms. *goes to look up phosgene*

 

In the Wikipedia it says phosgene can be created by a combination of carbon monoxide and chlorine gas exposed to sunlight.  So driving an old fashioned car near a swimming pool?  It’s apparently been in use in industry since the 19th century. 

 

Phosgene is the lethal ingredient in nerve agents. Shuts down your nerves, (hence the term nerve agent).

[Bombastinator]

16 minutes ago, CarlBar said:

 

Funny fact, i visited a chem company in the north of the UK during a collage course, (economics stupidly enough, long drive, ughh, bad memories), and i asked what the nastiest thing they used there was and what it was used for. Answer: Mustard gas. Used at the time as a precursor in the production of Viagra. 

 

Also it's less special steels as it is they need a fluoride oxide layer AFAIK.

 

 

Phosgene is the lethal ingredient in nerve agents. Shuts down your nerves, (hence the term nerve agent).

Things that are merely obnoxious can get horrifying in the right situation.  I was told a story once about a US experiment in the Viet Nam war where they attempted to make a mild irritant to use a bit like CS.  They mixed the active chemical in poison ivy with another material that causes things to be absorbed through the skin. The results were catastrophic.  It was not used past that experiment. 

[tridy]

On 3/12/2021 at 5:44 AM, CarlBar said:

 

Part of the issue based on a post upthread is that a lot of the stuff hosted there is hosted by third party companies acting on behalf of the businesses and that they aren't allways upto task and it isn't allways obvious that is so till your at a point where it takes quite a while to move to somthing better.

 

So, here is you lizard. It is not the fire, but "hosted there by third party companies acting on behalf of the business". Whoever made the decision to host it there did not do the proper homework, or found "a much better deal" and ignored some important aspects. Who is the monkey?

[Kilrah]

16 minutes ago, tridy said:

So, here is you lizard. It is not the fire, but "hosted there by third party companies acting on behalf of the business". Whoever made the decision to host it there did not do the proper homework, or found "a much better deal" and ignored some important aspects. 

Sure, but the whole point is that is often not the end customer, who cannot act on this or even necessarily know whether it's handled correctly. 

[CarlBar]

2 hours ago, Kilrah said:

Sure, but the whole point is that is often not the end customer, who cannot act on this or even necessarily know whether it's handled correctly. 

 

This, if your going 3rd party it's because you lack the expertise to know what your doing yourself and it's cheaper to get an outside expert in to look over things and handle matters. It's the same reason if you find yourself involved in a court case that you generally want a lawyer representing you. In general you don't know enough to do a good job so you get someone who does know enough to do it for you.

 

Ideally you'd know enough to know if someones any good at their job, but even that may be hard, especially if they get nasty and bury details in obscure places and otherwise flirt with the line between false advertising and good advertising. There's a reason the phrase "sexing up a product" exists.

[PineyCreek]

SBG1's considered a loss as well...

 

https://www.theregister.com/2021/03/22/ovh_sbg1_written_off_restoration_update/

[Elvara]

6 hours ago, PineyCreek said:

SBG1's considered a loss as well...

 

https://www.theregister.com/2021/03/22/ovh_sbg1_written_off_restoration_update/

Not really they won't restart it but it's moving the servers to SBG4 and RBX/GRA and then the rest of SBG1 will go to the future SBG5. The founder Octave Klaba has been pretty good with the work update on twitter.