Delta Canceled 650 Flights due to Server outage

[EvilLemur]

So today around 5:05 Am delta airlines experianced a computer outage and it affected flights scheduled for the moring, a power outage in Atlanta (where delta HQ is located) was affected and flights were delayed. 

 

7:30 a.m. ET UPDATE: Delta has published a waiver for customers traveling on Aug. 8

 

As of 3:40 Pm delta has operated 2,340 of 6,000 flights. systems are operational and flights have resumed 

 

So now, everyone has laughed at LTT for there video out a power outage and how he could spend so much and do so little, but look at delta a company that can afford to do it right and look at how they were affected.. 

 

so it makes you think... how much we depend other computer systems and the internet.

 

http://news.delta.com/video-ceo-apologizes-customers-says-delta-working-hard-recover

[Cela1]

Delta should really get a bigger UPS

[Overkilled]

25 minutes ago, Cela1 said:

Delta should really get a bigger UPS

wut?

[Thony]

Flying with them in a month....

[AlTech]

2 hours ago, Overkilled said:

wut?

The UPS he was referring to is an external backup Power Supply which provides temporary power in case of emergencies.

[AlTech]

17 minutes ago, Thony said:

Flying with them in a month....

GG. Hope that they won't screwup then.

[ZestyJalapeno]

Better a power outage, than a stupid 5-day strike stranding hundreds of thousands of commuters for a week. Looking at you, Southern Rail...

[AlTech]

10 minutes ago, Dogeystyle said:

Better a power outage, than a stupid 5-day strike stranding hundreds of thousands of commuters for a week. Looking at you, Southern Rail...

Air France also had a massive strike where over 40% of their flights got cancelled.

 

The flight attendants were complaining that they weren't being paid enough.

[KuJoe]

Thread title is misleading, it wasn't a single server outage it was a whole data center outage. Those of us who deal with disaster recovery plans for petabytes of data understand their pain.

[Tomsen]

1 hour ago, KuJoe said:

Thread title is misleading, it wasn't a single server outage it was a whole data center outage. Those of us who deal with disaster recovery plans for petabytes of data understand their pain.

They should still spread/duplicate their data to multiple sites to avoid a single-failure-point in their infrastructure.

[EvilLemur]

5 hours ago, KuJoe said:

Thread title is misleading, it wasn't a single server outage it was a whole data center outage. Those of us who deal with disaster recovery plans for petabytes of data understand their pain.

yeah i was gonna fix it but i had to go to work. lol 

[KuJoe]

7 hours ago, Tomsen said:

They should still spread/duplicate their data to multiple sites to avoid a single-failure-point in their infrastructure.

Should and can are two different things though. In a perfect world without restrictions I'm sure they would but I am not sure what regulations the TSA enforces. Speaking from experience though, replicating 20+ petabytes of data is not a cheap or easy task.

[Tomsen]

16 minutes ago, KuJoe said:

Should and can are two different things though. In a perfect world without restrictions I'm sure they would but I am not sure what regulations the TSA enforces. Speaking from experience though, replicating 20+ petabytes of data is not a cheap or easy task.

They absolutely can from a technology standpoint. I think it is much more of a economic concern in restructuring IT (this can be an issue, depending on how much is build around their previous setup), training employees, and potential implementation issues. Doubt TSA have any regulations in this regard, what did you have in mind, no replica of customer info?

 

If the setup is done right, it should be just as easy replicating 1 terabyte as it would of 20 petabytes. Cost is another issue, but I don't think downtime is cheap either.

[KuJoe]

1 hour ago, Tomsen said:

They absolutely can from a technology standpoint. I think it is much more of a economic concern in restructuring IT (this can be an issue, depending on how much is build around their previous setup), training employees, and potential implementation issues. Doubt TSA have any regulations in this regard, what did you have in mind, no replica of customer info?

 

If the setup is done right, it should be just as easy replicating 1 terabyte as it would of 20 petabytes. Cost is another issue, but I don't think downtime is cheap either.

I can't speak for Delta, but in order for my company to run an "active-active" setup between two data centers the replication can be no more than 5 seconds apart which pretty much limits us to building data centers in the same state based on the volume of data we generate during market hours (I can only imagine a 24x7 operation like an airline). We've thrown hundreds of millions of dollars into disaster recovery just in the years I've been with the company (albeit a big chunk of that was building data centers) and it's not a simple task and there is no silver bullet solution (trust me, if my company could just write a check and we'd have an "active-active" setup we definitely would in a heartbeat a lot of companies have claimed they could do it including EMC but in the end it didn't work). Sure Delta has a little under double the annual income that my company has ($926m compared to $501m), but like I said there is no magic bullet to drop into an existing infrastructure that I know of. If you're a start-up in 2016 you can easily build a resilient infrastructure to begin with or if you're Google you can rebuild everything from scratch because you're running mostly in house code.

 

Some of the biggest hurdles for my company are ones put in place by the SEC. I would imagine the TSA also has similar regulations in regards to client info (KYC-type requirements) since they basically do the same thing although the SEC's regulations are to prevent people from funding illegal activities while the TSA's goal is to prevent certain people or groups from traveling freely within the US, again I don't know so maybe the TSA is extremely lax compared to the SEC and if so I have a really easy solution that could have prevented this for roughly $10k per airport that Delta is located in.