OVH Cloud suffers major fire in Strasbourg datacenter

[IwishIcanFLighT]

Summary

 

Last night, one of OVH Cloud datacenter in Strasbourg had a major fire. At the time of writing (story is breaking), the situation is as follows:

• SBG1 is partially burnt: network room OK, 4 rooms destroyed, 8 are OK (update by CEO 11:40 local time)
• SBG2 is gone
• SBG3 is okay (update by CEO 11:20 local time)
• SBG4 is okay

Update:

The entire datacenter is currently shutdown out of precaution. A lot of French websites (including government ones) relying on this datacenter are unreachable since Wednesday. Rust servers lost some players stats that went in smoke.

OVH says it'll take a couple of weeks to restart the datacenter.

 

 

 

Quotes

From OVH Travaux

Quote

We are currently facing a major incident in our DataCenter of Strasbourg with a fire declared in the building SBG2.
Firefighters were immediately on the scene but could not control the fire in SBG2.
The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4.
If your production is in Strasbourg, we recommend to activate your Disaster Recovery Plan.
All our teams are fully mobilized along with the firefighters.
We will keep you updated as more information becomes available.

From  Octave Klaba (CEO):

My thoughts

 

Well, obviously,

 

BACKUP YOUR DATA.

 

I personally have a few VPS located in this datacenter (not SBG2 but SBG1, so I don't know how to feel about this yet).

Also, we sure will get a lot of Backblaze sponsor spot in the following days and week...

 

Sources

This is a french news story and the US is still waking up, so french only for now sorry:

https://www.lemonde.fr/societe/article/2021/03/10/a-strasbourg-un-important-incendie-sur-le-site-de-l-entreprise-ovh-classe-seveso_6072548_3224.html

https://www.leparisien.fr/faits-divers/strasbourg-un-important-incendie-sur-le-site-de-lentreprise-ovh-qui-abrite-des-serveurs-informatiques-10-03-2021-36SU6LXHHJF4BCZEOKNOAAKLVA.php

https://france3-regions.francetvinfo.fr/grand-est/bas-rhin/strasbourg-0/strasbourg-gros-incendie-dans-un-data-center-du-port-du-rhin-pas-de-blesse-1990768.html

 

[Tad Bittoomuch]

This is incredibly unfortunate.  My server was in SBG2 and is completely gone.

 

I do however have backups, so I still have all my data.

My condolences to those who had production and/or backup servers in SBG2

Edit:  A potential cause of the fire could be attributed to some defected power cables.  It seems the same day of the fire, a task was entered in their ticket tracking software to replace all the power supply cables in another datacentre.

http://travaux.ovh.net/?do=details&id=49017

It's possible they didn't get to them all in time ¯\_(ツ)_/¯

 

Press Release 1:
https://linustechtips.com/topic/1313943-ovh-cloud-suffers-major-fire-in-strasbourg-datacenter/?do=findComment&comment=14549259

 

Press Release 2:

https://linustechtips.com/topic/1313943-ovh-cloud-suffers-major-fire-in-strasbourg-datacenter/?do=findComment&comment=14551368

 

Edit: It wasn't the power cables.  At this stage it looks like the failure of 2 UPSs was the cause of the blaze.

https://www.ovh.com/fr/images/sbg/Octave-Klaba-speaking-en-vid.mp4

[Sauron]

They must have upgraded to 11 series Intel CPUs  

 

I would assume OVH had a full backup of this datacenter somewhere else... otherwise it would be pretty bad

[Kilrah]

My brother has an osteopathy practice and the records management software they use was apparently hosted there... The software provider is a small company that has a poor record of proper data management so I hope for him they have backups but I wouldn't be so sure... of course we were in the middle of planning a transition to another provider due to said issues with these guys 

 

Hopefully OVH's backup solutions are distributed enough and the guys were using them...

[Tad Bittoomuch]

17 minutes ago, Sauron said:

They must have upgraded to 11 series Intel CPUs  

 

I would assume OVH had a full backup of this datacenter somewhere else... otherwise it would be pretty bad

I can't find it now, but I remember somewhere in the OVH Terms of Service, there's a clause in there about how OVH is not responsible for any user's data - meaning, if users haven't got backups of their environments and they were hosted in SBG2 then they're gone.

I'll attach a source if I find it

Edit: here we go:

This is from their 2014 TOS:

Quote

If the performance of this Agreement, or of any of OVH’s obligations under such Agreement, is barred, limited or disrupted due to a fire, an explosion, a failure in the transmission networks, the collapse of the facilities, an epidemic, an earthquake, a flood, a power failure, a war, an embargo, a statute, an injunction, a request or demand from a government, a strike, a boycott, a withdrawal of authorization from the telecommunications operator, or any other circumstances beyond OVH’s reasonable control... [continued]

...

... OVH does not offer any backup for the hosted data. It is therefore the Customer’s responsibility to take all required steps in order to back up his/its data in the event of a loss or deterioration of such data for any reason whatsoever, including reasons not explicitly mentioned hereunder

This is from their TOS you are shown when you order a server:

Quote

OVH shall not be liable on any grounds whatsoever to the extent permitted by law for any direct, indirect, secondary or consequential loss or loss of income in the circumstances described below:

Quote

3.5.1.If the performance of this Agreement, or of any of OVH’s obligations under such Agreement, is barred, limited or disrupted due to a fire, an explosion, a failure in the transmission networks, the collapse of the facilities, an epidemic, an earthquake, a flood, a power failure, a war, an embargo, a statute, an injunction, a request or demand from a government, a strike, a boycott, a withdrawal of authorization from the telecommunications operator, or any other circumstances beyond OVH’s reasonable control...

Quote

3.11. The Customer is aware and acknowledges that OVH does not offer any backup for the Customer’s hosted data. It is therefore the Customer’s sole responsibility to take all required steps in order to back up his/its data in the event of a loss or deterioration of such data for any reason whatsoever, including reasons not explicitly mentioned hereunder.

 

 

[Kilrah]

Usually when you get a simple website hosting that's backed up, but when you get a VPS / bare metal machine then backups are on you by default, however providers typically provide a simple backup solution in their infrastructure and provided you subscribe to that it should do the job...

[Tad Bittoomuch]

They issued a press release:

 

Spoiler

 

[Sauron]

24 minutes ago, Tad Bittoomuch said:

I can't find it now, but I remember somewhere in the OVH Terms of Service, there's a clause in there about how OVH is not responsible for any user's data - meaning, if users haven't got backups of their environments and they were hosted in SBG2 then they're gone.

I'll attach a source if I find it

Edit: here we go:

This is from their 2014 TOS:

This is from their TOS you are shown when you order a server:

 

Oh, RIP

[leadeater]

1 hour ago, Tad Bittoomuch said:

there's a clause in there about how OVH is not responsible for any user's data - meaning, if users haven't got backups of their environments and they were hosted in SBG2 then they're gone.

That's typical of any hosting provider, it's a supply of services not disaster protection plan with it. So unless the service you are paying for is Geo-Replicated, which is generally a premium option upgrade, it's on the customers to implement data protection. And even then replication is not backup, neither are snapshots and replicated snapshots that live in the primary storage bucket/array.

 

Even if you are using a DRaaS offering the service itself does not include DR or backups, it's a service you use to implement in to your architecture design to create a DR strategy and operational procedure.

 

This is the difference between High Availability (HA) and High Resiliency (HR), what you're getting is only HR unless you have purchased an HA option like Geo-Replication, which still excludes backup.

 

It's these specific, but very important, details that often get overlooked and not properly considered with the current trend to move services in to the Cloud and this issue is compounded by Software as a Service and Platform as a Service offerings as you have no idea if DR and backups have been architected in to them at all and worse there may be no way to integrate in with backup software or some other process to extract data out regularly for backups. Sometimes things are cheaper because you are functionally getting an inferior service compared to what you had before.

[leadeater]

55 minutes ago, Kilrah said:

Usually when you get a simple website hosting that's backed up

Generally I have not seen this, at best I've seen is website and file/content versioning utilizing snapshots on the backend but the data all lives together so it only protects you from non infrastructure disruptive events, so a fire like this you would not be protected from as the backups would be destroyed.

[givingtnt]

Famous last words.

Been down ever since

[rcmaehl]

Looks like they had a firewall issue

Should have enabled alerts for lp0 on fire

Looks like the intern found the firewire cable

 

Okay okay I'll stop

[sub68]

Why is this perfect.

The LTT fourm actually went down

[D13H4RD]

The servers for TDU: Project Paradise were in SBG2

 

Well then....I guess no TDU online for a few weeks.....

[BuckGup]

Should have spent the extra money on a halon suppression system.

[wanderingfool2]

From OVH's website under "Maximum Physical Security"

Quote

Rooms fitted with smoke detection systems

Apparently they missed the fire suppression portion.  Honestly though, it would be interesting to see an analysis of where the fire started and how it was able to spread so much.  In the datacenter's I've been in, they all have been outfitted with systems that would allow damage to be very contained (and in some cases have it starve the fire of oxygen)

 

At the very least, I would have thought they would have had better fire suppression system that would have prevented the spread to multiple datacenter rooms.

[AnonymousGuy]

I thought the whole point of datacenters existing is they're supposed to be nearly bullet proof  to everything but large scale natural disaster.  

 

Is this where we learn their cheap shipping container whatever the fuck infrastructure is trash? (but at least it's hopefully cheap)

[StDragon]

5 minutes ago, wanderingfool2 said:

At the very least, I would have thought they would have had better fire suppression system that would have prevented the spread to multiple datacenter rooms.

The walls should be solid concrete for data center room isolation.

[Vanderburg]

Proof that "cloud" is a meaningless marketing term.

[StDragon]

1 hour ago, Vanderburg said:

Proof that "cloud" is a meaningless marketing term.

"Cloud" is someone else's server hosted off premises.

[small guy64]

and this is why you have backup and secondary locals, cloud is nether of those things. 

[Vanderburg]

18 minutes ago, StDragon said:

"Cloud" is someone else's server hosted off premises.

Like I said...

[Kisai]

12 hours ago, IwishIcanFLighT said:

 

My thoughts

 

Well, obviously,

 

BACKUP YOUR DATA.

 

I personally have a few VPS located in this datacenter (not SBG2 but SBG1, so I don't know how to feel about this yet).

Also, we sure will get a lot of Backblaze sponsor spot in the following days and week...

 

 

Keep in mind that OVH is one of the large, significant ISP's that host illegal content protected by cloudflare. So it would be interesting for cloudflare to report what % of their customers sites are now permanently gone.

 

[Beerzerker]

Man - Look at all the data coming out of the building!
 

Seriously I do hope no one was injured or worse.
 

[GDRRiley]

1 minute ago, Beerzerker said:

Seriously I do hope no one was injured or worse.

no one has been hurt.
just lots of lost data