OVH Cloud suffers major fire in Strasbourg datacenter

[Beerzerker]

Just now, GDRRiley said:

no one has been hurt.
just lots of lost data

You can say that again..... And everyone can see where it went.

[Kisai]

4 minutes ago, Beerzerker said:

You can say that again..... And everyone can see where it went.

They let the magic smoke out that data is stored in /s

[Heliian]

5 hours ago, BuckGup said:

Should have spent the extra money on a halon suppression system.

Halon is pretty scarce these days but there are good alternatives.  

However, even the top of the line systems can only delay a fierce fire. 

[JamesTheFolf]

13 hours ago, IwishIcanFLighT said:

Summary

 

Last night, one of OVH Cloud datacenter in Strasbourg had a major fire. At the time of writing (story is breaking), the situation is as follows:

• SBG1 is partially burnt: network room OK, 4 rooms destroyed, 8 are OK (update by CEO 11:40 local time)
• SBG2 is gone
• SBG3 is okay (update by CEO 11:20 local time)
• SBG4 is unknown (but is hopefully fine)

A lot of French websites (including government ones) relying on this datacenter are unreachable this morning.

 

 

 

Quotes

From OVH Travaux

From  Octave Klaba (CEO):

My thoughts

 

Well, obviously,

 

BACKUP YOUR DATA.

 

I personally have a few VPS located in this datacenter (not SBG2 but SBG1, so I don't know how to feel about this yet).

Also, we sure will get a lot of Backblaze sponsor spot in the following days and week...

 

Sources

This is a french news story and the US is still waking up, so french only for now sorry:

https://www.lemonde.fr/societe/article/2021/03/10/a-strasbourg-un-important-incendie-sur-le-site-de-l-entreprise-ovh-classe-seveso_6072548_3224.html

https://www.leparisien.fr/faits-divers/strasbourg-un-important-incendie-sur-le-site-de-lentreprise-ovh-qui-abrite-des-serveurs-informatiques-10-03-2021-36SU6LXHHJF4BCZEOKNOAAKLVA.php

https://france3-regions.francetvinfo.fr/grand-est/bas-rhin/strasbourg-0/strasbourg-gros-incendie-dans-un-data-center-du-port-du-rhin-pas-de-blesse-1990768.html

 

This is a terrible accident, and I hope that OVH can get to the bottom of it. I also hope that no one lost any data forever, but it is entirely possible. This is why companies should have more than just the standard 3-2-1 rule for their data.

[StDragon]

28 minutes ago, Heliian said:

However, even the top of the line systems can only delay a fierce fire. 

I'm still blown away at how out of control the fire got.

 

Fire needs 3 things - heat, fuel, and oxygen. Given most computer hardware is made of metal and plastic, I can't image there being that much plastic and cabling to fuel a fire of that magnitude. The damage looked like that of a car fire; complete with being hot enough to melt metal (from what I can see in the photos)!

 

 

[Heliian]

55 minutes ago, StDragon said:

I'm still blown away at how out of control the fire got.

 

Fire needs 3 things - heat, fuel, and oxygen. Given most computer hardware is made of metal and plastic, I can't image there being that much plastic and cabling to fuel a fire of that magnitude. The damage looked like that of a car fire; complete with being hot enough to melt metal (from what I can see in the photos)!

 

 

I imagine the place being stacked with equipment and cabling everywhere, office furniture, even added in walls and such.  Could have had a lot of ventilation which fueled the fire. I'm not familiar with their setup. 

 

In any case, it is really crappy if you've lost work in this fire but glad no reported injuries. 

 

 

 

[Kisai]

1 hour ago, StDragon said:

I'm still blown away at how out of control the fire got.

 

Fire needs 3 things - heat, fuel, and oxygen. Given most computer hardware is made of metal and plastic, I can't image there being that much plastic and cabling to fuel a fire of that magnitude. The damage looked like that of a car fire; complete with being hot enough to melt metal (from what I can see in the photos)!

 

 

 

Maybe it was the generator room or battery backups that caught fire.

 

Honestly, this is one of those cases where it sounds like fire separations were insufficient to contain the fire.

 

Or maybe they over-provisioned power, and it literately just caught fire from that.

[Jaydnems]

Dang that sucks, that must be a lot lost

[Bombastinator]

16 hours ago, IwishIcanFLighT said:

Summary

 

Last night, one of OVH Cloud datacenter in Strasbourg had a major fire. At the time of writing (story is breaking), the situation is as follows:

• SBG1 is partially burnt: network room OK, 4 rooms destroyed, 8 are OK (update by CEO 11:40 local time)
• SBG2 is gone
• SBG3 is okay (update by CEO 11:20 local time)
• SBG4 is unknown (but is hopefully fine)

A lot of French websites (including government ones) relying on this datacenter are unreachable this morning.

 

 

 

Quotes

From OVH Travaux

From  Octave Klaba (CEO):

My thoughts

 

Well, obviously,

 

BACKUP YOUR DATA.

 

I personally have a few VPS located in this datacenter (not SBG2 but SBG1, so I don't know how to feel about this yet).

Also, we sure will get a lot of Backblaze sponsor spot in the following days and week...

 

Sources

This is a french news story and the US is still waking up, so french only for now sorry:

https://www.lemonde.fr/societe/article/2021/03/10/a-strasbourg-un-important-incendie-sur-le-site-de-l-entreprise-ovh-classe-seveso_6072548_3224.html

https://www.leparisien.fr/faits-divers/strasbourg-un-important-incendie-sur-le-site-de-lentreprise-ovh-qui-abrite-des-serveurs-informatiques-10-03-2021-36SU6LXHHJF4BCZEOKNOAAKLVA.php

https://france3-regions.francetvinfo.fr/grand-est/bas-rhin/strasbourg-0/strasbourg-gros-incendie-dans-un-data-center-du-port-du-rhin-pas-de-blesse-1990768.html

 

I have a friend who used to claim the moral of every story ever was “always make backups”

[CarlBar]

5 hours ago, Kisai said:

They let the magic smoke out that data is stored in /s

 

And once the magic smoke comes out there's no getting it back in.

 

5 hours ago, Heliian said:

Halon is pretty scarce these days but there are good alternatives.  

However, even the top of the line systems can only delay a fierce fire. 

 

Depends on the fire, depends on the system. A Halon or Halon like system can kill any fire in it's room provided some explosion or the like doesn't put holes in the walls and the system can be fully automated. A fire can't burn without oxidant. The issue is that kind of system is expensive, requires specific setup, and considerations and outside of certain heavy industry and military environments is potentially more dangerous to human life than a fire in heavily trafficked area's. Thats why your average Joe public doesn't run into them much.

[Elvara]

That fire is pretty impressive and that building is pretty recent 2017 I believe, I use OVH but I've not been impacted but I'm still very curious on how or what started the fire, I remember seeing how they were testing different way to reduce the cost of cooling, like hollow building to have more airflow and also Strasbourg-1 was build using container after a long testing period and a deployment in Roubaix-4.

 

From what I see the building that caught fire used their tower design with the hollow part, it's the middle building in that rendering. Maybe we'll know more at some point.

 

[leadeater]

6 hours ago, Heliian said:

Halon is pretty scarce these days but there are good alternatives.  

However, even the top of the line systems can only delay a fierce fire. 

 

5 hours ago, StDragon said:

I'm still blown away at how out of control the fire got.

 

Fire needs 3 things - heat, fuel, and oxygen. Given most computer hardware is made of metal and plastic, I can't image there being that much plastic and cabling to fuel a fire of that magnitude. The damage looked like that of a car fire; complete with being hot enough to melt metal (from what I can see in the photos)!

As part of our facilities upgrades we're going with a hypoxic environmental system for fire suppression, that means the atmosphere does not support fire (flame). Down side to that is you have to have even more restricted access controls and health and safety forms and training to be allowed to go in to the room. You can still breath but it's equivalent to being in very high atmosphere, no everyone can handle being in that type of environment too.

 

Fire can still happen though if it become self oxidizing, or something similar to that. 

[wanderingfool2]

11 minutes ago, leadeater said:

 

As part of our facilities upgrades we're going with a hypoxic environmental system for fire suppression, that means the atmosphere does not support fire (flame). Down side to that is you have to have even more restricted access controls and health and safety forms and training to be allowed to go in to the room. You can still breath but it's equivalent to being in very high atmosphere, no everyone can handle being in that type of environment too.

 

Fire can still happen though if it become self oxidizing, or something similar to that. 

Now that sounds pretty cool.  Would love to be able to visit one of those kind of controlled datacenters in my lifetime.

 

I'm still amazed at how far the fire was able to spread.  If they are making claims of smoke detection systems, I would have thought they would have had better protection in place from it spreading to multiple rooms.  Although speaking from experience, computers have an amazing large amount of burnable material (was sitting next to a test PC once when it began billowing out smoke)

[leadeater]

4 minutes ago, wanderingfool2 said:

I'm still amazed at how far the fire was able to spread.  If they are making claims of smoke detection systems, I would have thought they would have had better protection in place from it spreading to multiple rooms.  Although speaking from experience, computers have an amazing large amount of burnable material (was sitting next to a test PC once when it began billowing out smoke)

If I had to pick a cause then my money would be on UPS battery fire. Those are extremely dangerous, self oxidizing, and spreads extremely quickly. No amount of smoke detection and 24/7 on site monitoring will prevent something like that, only proper containment built to handle a battery fire.

 

So the fact nobody was able to stop it in time and it spread so quickly I'm personally looking at batteries.

[Tad Bittoomuch]

OVH issued another press release

 

Page 1:

Spoiler

 

Page 2:

Spoiler

 

[tridy]

Someone sometime said once:

There Is No Cloud It's Just Someone Else's Computer

 

If you are the architect of the company who lost the data - it is your fault. Know the lizards that can attack - fires, floods, thunders, meteorites, cable monkeys, and be prepared.

[Sir Asvald]

The Server hardware must be LIT!

 

/s

[RejZoR]

6 hours ago, Elvara said:

That fire is pretty impressive and that building is pretty recent 2017 I believe, I use OVH but I've not been impacted but I'm still very curious on how or what started the fire, I remember seeing how they were testing different way to reduce the cost of cooling, like hollow building to have more airflow and also Strasbourg-1 was build using container after a long testing period and a deployment in Roubaix-4.

 

From what I see the building that caught fire used their tower design with the hollow part, it's the middle building in that rendering. Maybe we'll know more at some point.

 

I'm no firefighter, but when building is shaped like a chimney, it'll act like a chimney when on fire. And sole purpose of chimneys is to actually induce fresh air to the burning material for as ideal combustion as possible. Kinda the opposite of what you want in a building itself...

 

Also whole thing looks like someone thought it would be cool to cheaply make a building out of I don't know, decomissioned shipping containers or something? I'm fairly certain data centers are built in very specific ways to mitigate such things specifically, makes me wonder how much of it was done here... And because it's expensive electronic equipment, fire supression has to be non destructive. Aka suffocation of fire using CO2 or venting of internal air instead of water sprinklers. Obviously.

[Bombastinator]

4 hours ago, RejZoR said:

I'm no firefighter, but when building is shaped like a chimney, it'll act like a chimney when on fire. And sole purpose of chimneys is to actually induce fresh air to the burning material for as ideal combustion as possible. Kinda the opposite of what you want in a building itself...

 

Also whole thing looks like someone thought it would be cool to cheaply make a building out of I don't know, decomissioned shipping containers or something? I'm fairly certain data centers are built in very specific ways to mitigate such things specifically, makes me wonder how much of it was done here... And because it's expensive electronic equipment, fire supression has to be non destructive. Aka suffocation of fire using CO2 or venting of internal air instead of water sprinklers. Obviously.

I watched a video recently about shipping container buildings that was not very positive.    Apparently they make pretty poor building material.  The spaces created are extremely claustrophobic when actually kitted out and they’re only actually strong in very specific orientations and configurations which people tend not to use.  The result is they’re often basically extremely expensive steel paneling.  Some firefighting techniques and technologies haven’t changed much since Roman times.  Fire fighters actually sometimes still carry spears for very specific uses.  There isn’t a lot of research done on fires which makes what is done incredibly valuable.  There was a town in Canada that was control burned in the 1950’s the data from which is still massively important world wide.  Stuff can be extremely non intuitive.  Two technologies I remember reading about is covering a wall with paper can actually make it more fireproof, (which produced Sheetrock)  and one of the most fireproof things in a house is a dirty disposable diaper.  (Which produced modern document safes) It’s possible they simply couldn’t do much with a building like this.  I suspect specifics will emerge over time.  

Edited March 11, 2021 by Bombastinator

[soldier_ph]

Not even the cloud itself is save. That building has an interesting facade design. Hope the destroyed Hardware and especially the Drives are secured for Cyber/Data Security. 

[GoodBytes]

3 hours ago, soldier_ph said:

Not even the cloud itself is save. That building has an interesting facade design. Hope the destroyed Hardware and especially the Drives are secured for Cyber/Data Security. 

And people were making fun of me when I said cloud the cloud.... Not so stupid now!!! :sigh:

 

[CarlBar]

18 hours ago, tridy said:

Someone sometime said once:

There Is No Cloud It's Just Someone Else's Computer

 

If you are the architect of the company who lost the data - it is your fault. Know the lizards that can attack - fires, floods, thunders, meteorites, cable monkeys, and be prepared.

 

Part of the issue based on a post upthread is that a lot of the stuff hosted there is hosted by third party companies acting on behalf of the businesses and that they aren't allways upto task and it isn't allways obvious that is so till your at a point where it takes quite a while to move to somthing better.

[Elvara]

20 hours ago, RejZoR said:

I'm no firefighter, but when building is shaped like a chimney, it'll act like a chimney when on fire. And sole purpose of chimneys is to actually induce fresh air to the burning material for as ideal combustion as possible. Kinda the opposite of what you want in a building itself...

 

Also whole thing looks like someone thought it would be cool to cheaply make a building out of I don't know, decomissioned shipping containers or something? I'm fairly certain data centers are built in very specific ways to mitigate such things specifically, makes me wonder how much of it was done here... And because it's expensive electronic equipment, fire supression has to be non destructive. Aka suffocation of fire using CO2 or venting of internal air instead of water sprinklers. Obviously.

Well the datacenter in container is apparently nothing new and it's used from what I saw on google well even google works or have worked with it https://www.youtube.com/watch?v=zRwPSFpLX8I but it's not the building that caught on fire.

But yeah I didn't think of it like that the building maximizing airflow, would also maximize fire propagation and smoke. I never witness computers starting a fire, there's quite a lot of metal, that's why I was surprised at the size of the fire.
I work in the hotel industry and our fire safety is pretty hardcore from fire detection to what kind of paint or fabric we can use in rooms or corridors also it's mostly fire retardant materials. I know almost everything still burn at some point but I'm surprised how industrial building burn that big specially were they don't stock flammable materials and the interior is most of the time bare walls.

[leadeater]

2 hours ago, Elvara said:

Well the datacenter in container is apparently nothing new and it's used from what I saw on google well even google works or have worked with it

We have a container and they can be well made and designed, just depends on what your end goal is. Ours is a DR + Main site so race to bottom dollar isn't the purpose of it, it's our crap hitting the fan catcher so it has to work. It's built to be submerged under water if that sort of disaster happens and has isolated air to water to air cooling, which is how it gets to be water resistant. Also has Halon Gas fire suppression system.

 

It's quite old now, we had to slightly lift it with a crane and recoat the bottom to ensure it stayed weather/rust resistant.

 

Every 6 months we do an uncharged (Halon release valve closed) fire suppression test with real flame and smoke trigger and make sure the system properly activates

[Bombastinator]

3 hours ago, Elvara said:

Well the datacenter in container is apparently nothing new and it's used from what I saw on google well even google works or have worked with it https://www.youtube.com/watch?v=zRwPSFpLX8I but it's not the building that caught on fire.

But yeah I didn't think of it like that the building maximizing airflow, would also maximize fire propagation and smoke. I never witness computers starting a fire, there's quite a lot of metal, that's why I was surprised at the size of the fire.
I work in the hotel industry and our fire safety is pretty hardcore from fire detection to what kind of paint or fabric we can use in rooms or corridors also it's mostly fire retardant materials. I know almost everything still burn at some point but I'm surprised how industrial building burn that big specially were they don't stock flammable materials and the interior is most of the time bare walls.

My understanding of data center containers is they’re not work spaces.  They’re filled with machinery and are more a single device in the shape of a shipping container than a structure.  Much less merely part of a structure partially made from shipping containers.  Thinking of one as the other is nothing but sympathetic magic.