Azure Facility in South Central US Down.

[JacobFW]

Not sure if I'm the only one that cares or not, but I can't do my job right now so screw it.

 

From https://azure.microsoft.com/en-us/status/  (which now also appears to be down)

Quote

 

 Warning Multiple Services - South Central US - Investigating

Starting at 09:29 UTC on 04 Sep 2018 a subset of customers in South Central US may experience difficulties connecting to resources hosted in this region. Engineers have isolated an issue with cooling in one part of the data center, which caused a localized spike in temperature, as the preliminary root-cause. Automated data center procedures to ensure data and hardware integrity went into effect when temperatures hit a specified threshold and critical hardware entered a structured power down process. The impact to the cooling system has been isolated and is in the process of being mitigated. Engineers are continuing to work towards restoration of services. The next update will be provided at 14:00 UTC or as events warrant.

 

 

 

[ItsMitch]

Quote

Engineers have isolated an issue with cooling in one part of the data center, which caused a localized spike in temperature

intel cpus strike again

 

 

[smount92]

They're having an issue with Office365 services as well.

 

status.office365.com

[schwellmo92]

Ouchies. Not quite following how a localised spike in temperatures brings down the entire data centre, maybe they’re understating it?

 

You would think they’d have redundancy up the whazoo and a localised issue shouldn’t cause everything to go down.

[JacobFW]

2 minutes ago, schwellmo92 said:

Ouchies. Not quite following how a localised spike in temperatures brings down the entire data centre, maybe they’re understating it?

 

You would think they’d have redundancy up the whazoo and a localised issue shouldn’t cause everything to go down.

They're in Arizona

[ZestyJalapeno]

4 minutes ago, schwellmo92 said:

Ouchies. Not quite following how a localised spike in temperatures brings down the entire data centre, maybe they’re understating it?

 

You would think they’d have redundancy up the whazoo and a localised issue shouldn’t cause everything to go down.

"localized spike in temperature, as the preliminary root-cause"

 

"critical hardware entered a structured power down process"

 

Shutting down in case of potential, actual failure of critical server modules is the first step of disaster mitigation. A day's downtime because of failed cooling is a lot better than clients losing their data.

 

[goodtofufriday]

North east most services are Okay. But authentication services are being iffy. Azure billing portal is completely down.

[Radium_Angel]

4 minutes ago, goodtofufriday said:

North east most services are Okay. But authentication services are being iffy. Azure billing portal is completely down.

And people say working in the "cloud" is the future.

Pffft

[JacobFW]

28 minutes ago, Dogeystyle said:

"localized spike in temperature, as the preliminary root-cause"

 

"critical hardware entered a structured power down process"

 

Shutting down in case of potential, actual failure of critical server modules is the first step of disaster mitigation. A day's downtime because of failed cooling is a lot better than clients losing their data.

 

Agreed, doesn't make it any less of a pain in the ass though.

22 minutes ago, Radium_Angel said:

And people say working in the "cloud" is the future.

Pffft

I'll give them credit.  This is the first major issue we've had with them.  I imagine Microsoft could do a better job, but this also goes to show why you shouldn't put all your eggs in one basket.  We're looking into adding some redundancy so we're not totally reliant on them.

[ShawnTD]

And to think Microsoft is increasing their CAL's pricing with server 19 to push people to azure...?

[samcool55]

1 hour ago, Radium_Angel said:

And people say working in the "cloud" is the future.

Pffft

Yes and no, it will have a place in the future. But it won't replace everything.

Also the big boys are pushing cloud stuff hard because it gives them a constant flow of money like office subscriptions.

And, if you do go to the cloud, you are depending a lot more on tools your cloud provider provides to migrate stuff.

AD for example, going from on-premise to cloud, no big deal, the other way, almost impossible.

 

They want everybody to be in the cloud because once you are there, which is getting easier and easier to do, it's stupidly difficult to go back.

[Radium_Angel]

57 minutes ago, samcool55 said:

Yes and no, it will have a place in the future. But it won't replace everything.

Also the big boys are pushing cloud stuff hard because it gives them a constant flow of money like office subscriptions.

And, if you do go to the cloud, you are depending a lot more on tools your cloud provider provides to migrate stuff.

AD for example, going from on-premise to cloud, no big deal, the other way, almost impossible.

 

They want everybody to be in the cloud because once you are there, which is getting easier and easier to do, it's stupidly difficult to go back.

I am fortunate I have a say-so in my office about this. And that say-so is "not alone no, but HELL no!"

[Gullerback]

Wonder if this is related to BMO.  The entire BMO system went down today.  Still down.

[Drak3]

1 hour ago, samcool55 said:

They want everybody to be in the cloud because once you are there, which is getting easier and easier to do, it's stupidly difficult to go back

It's not that hard to go back. And for some of us, becoming cloud dependant isn't even in the realm of possibility just because we constantly use resource heavy programs or file sizes too large to effectively store remotely.

[JacobFW]

Starting to get some services back now.

 

 

[samcool55]

2 hours ago, Drak3 said:

It's not that hard to go back. And for some of us, becoming cloud dependant isn't even in the realm of possibility just because we constantly use resource heavy programs or file sizes too large to effectively store remotely.

Oh it is, good luck getting your PAAS and SAAS stuff that runs on azure back on-prem or even migrated to another cloud provider like AWS. They are working to make migrations between AWS and azure easier but it's going to take years before you can easily switch cloud providers.

 

And there will always be cases where cloud isn't an option, on-prem won't go away but it's just that, some companies will go cloud and some won't, cloud is here to stay but it won't take over.

[mr moose]

5 hours ago, JacobFW said:

Agreed, doesn't make it any less of a pain in the ass though.

I'll give them credit.  This is the first major issue we've had with them.  I imagine Microsoft could do a better job, but this also goes to show why you shouldn't put all your eggs in one basket.  We're looking into adding some redundancy so we're not totally reliant on them.

 

Given the size of operations of google, Azure and AWS they do a bloody good job of keeping everything working 24/7.

[schwellmo92]

13 hours ago, Dogeystyle said:

"localized spike in temperature, as the preliminary root-cause"

 

"critical hardware entered a structured power down process"

 

Shutting down in case of potential, actual failure of critical server modules is the first step of disaster mitigation. A day's downtime because of failed cooling is a lot better than clients losing their data.

 

If there’s a “localised” spike it should shut down the necessary resources to protect itself. A localised issue shouldn’t have roll-on effects to the entire data center.

[leadeater]

But the cloud is immune to failures, that's what they keep saying. It's so resilient it'll never go down....

[Rune]

Fire is a pretty large spike in localized temperature.

[schwellmo92]

Hmm I just had a read of the Azure outage page again and it just mentions "cooling" so i'll give them the benefit of the doubt and assume it was a significant portion/all of their cooling systems in which sense it makes sense to take it down.

 

What is not cool though, is all of the "global" services that were affected (AAD/Intune/etc).

[JacobFW]

Forgot, I had wanted to post their update.