Poor Grammar Caused Amazon's Outage This Week. Sad!

[Guest Kloaked]

STAY IN SCHOOL, KIDS!

 

You more than likely heard about the server outage at Amazon this week. The outage was caused by a typo in a command that was entered by an Amazon employee - the command was intended to only take down a small amount of servers, presumably for maintenance, but instead took the command as a "KILL THEM! KILL THEM ALL!" command.

 

Quote

 

In an online statement, Amazon described the circumstances of the disruptive typo this way:

"The Amazon Simple Storage Service (S3) team was debugging an issue causing the S3 billing system to progress more slowly than expected. At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process.

"Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended."
 

 

 

The servers that were unintentionally taken down also serve as the backbone for popular internet retailers as well as communication applications, such as Slack. So those too were also experiencing issues.

 

Quote

The Wall Street Journal reported that the outage "cost companies in the S&P 500 index $150 million, according to Cyence Inc., a startup that specializes in estimating cyber-risks. Apica Inc., a website-monitoring company, said 54 of the internet's top 100 retailers saw website performance slow by 20% or more."

 

In order to combat poor grammar and typing skills, Amazon has implemented better training and rules for the employees responsible for this hardware.

 

Quote

Amazon said it has changed its protocol for the routine, temporary removal of servers from its system so that server capacity is taken offline more slowly, among other safeguards.

 

So what have we learned? Number one: knowing how to English (or in this case whichever syntax the servers there respond to). Number two: if you don't follow the first lesson, you will most likely become a meme at work - known as the guy who made a typo and potentially cost retailers millions of dollars and slowed down chat and task managers for millions of users. That is, if you didn't get fired.

[SansVarnic]

Wow. Just. Wow.

[NumLock21]

Retoast

[Guest Kloaked]

1 minute ago, NumLock21 said:

Retoast

If you're talking about the thread that brought up the outage, this is not a repost.

[SCHISCHKA]

its not grammar its poor maintenance procedure

[NumLock21]

Just now, Kloaked said:

If you're talking about the thread that brought up the outage, this is not a repost.

It was mention the server was taken down by a single character or something, can't remember what it was.

[LegendKillerRG]

Typo..

[Guest Kloaked]

2 minutes ago, NumLock21 said:

It was mention the server was taken down by a single character or something, can't remember what it was.

Well let me know when you find it.

 

3 minutes ago, SCHISCHKA said:

its not grammar its poor maintenance procedure

This could have been avoided should the employee know basic language skills. I mean come on.

[zMeul]

- are you sure you want to format drive c: ? [Yes/No]

- [yes]

- formatting 100%

- wait .. what?!

[SCHISCHKA]

7 minutes ago, Kloaked said:

This could have been avoided should the employee know basic language skills. I mean come on.

have you ever looked at a terminal command? it was human error. they will asses it and put in place measures so it doesn't happen again. You want to bash the employee for making a mistake that anyone could make; I would criticize Amazon for allowing one single employee the authority to take down so many web services

[Guest Kloaked]

10 minutes ago, SCHISCHKA said:

have you ever looked at a terminal command? it was human error. they will asses it and put in place measures so it doesn't happen again. You want to bash the employee for making a mistake that anyone could make; I would criticize Amazon for allowing one single employee the authority to take down so many web services

The guy should be fired for not knowing how to properly type out a word and/or numbers. Send him back to school.

 

 

 

 

 

 

 

 

I am not being serious, just for the record.

[Suika]

I'm curious as to what command would take debugging a small chunk of devices to the majority of them.

 

Anyway, this is partly why I practice typing in the entire command (or at least, type the command long enough that no other command could be confused with it). Even if it contains a typo, IOS will reject it. This guy probably tried shortening the command a bit too far and accidentally fat fingered the wrong key.

[Granular]

Don't drink and sudo

[Misanthrope]

Not sure if a syntax error can qualify as "grammar". Yes we call them programming "languages" but there are differences between "grammar" which is "What we agreed should be the correct way cause fuck you, we know better" kind of rules vs "What we know has to happen to correctly parse commands in the correct order otherwise it will literally not compute"

 

Grammar is consensus and more than a bit of tradition going into it.

 

Syntax is deciding the best order and keeping to it since everything depends on the assumption that you'll follow the syntax.

 

If I say "I, in the summer nights, walking do enjoy!" most people would need to take pause but then would understand and call me a stupid fucking wetback stealing "MUH JOBS!" or something, but the point is that most people would still understand. Whereas a command that is written differently than what the correct syntax well computers simply will do what told resulting in unforeseen consequences, specially if the error got past what the language devs would detect as bad syntax and refuse to run.

[Guest]

AWS - "I know, lets trust our employee's so much, when they enter a command that causes an entire service outage, we don't question it" 

 

 

[Misanthrope]

16 minutes ago, AUniqueName said:

AWS - "I know, lets trust our employee's so much, when they enter a command that causes an entire service outage, we don't question it" 

Actually...is not as bad of a practice as you might think. This one is public, and with very high stakes so most people would laugh and say "oh what an idiot!" and "Fire him!" and stuff like that.

 

But I've been working for a big company for the last 2 years and I see this kind of stuff happen often, my department even caused an outage once. Like a big one that took a few hours of delayed trips and a bunch of money (we work logistics). But you basically acknowledge it and forget all drama for that moment and first figure out a solution. We had managers call out for blood, we had negative views from other parts of our company but our manager just said "It was a mistake and overlook, we'll work with them" and kept all of us working. 

 

The thing is that there is no such thing as a perfect employee and most companies only need you to be technically capable up to a certain degree because 1) They can't afford to recruit all positions with incredibly knowledgeable individuals as full time staff, that shit is prohibitively expensive and 2) They are interested in people they know they can develop into incredibly knowledgeable individuals in house and retain them. Most of us usually get frustrated and brush off "career paths" as HR fucking nonsense but I've seen newbies come in, fuck up everything and be a dead weight on their teams only to gradually work themselves into being indispensable. Most people that have a good attitude and talent to begin with only need the vote of confidence and good managers know this and can overlook honest mistakes, they're more interested in the type of person that would own up to them, fix them and learn from it. 

[sof006]

Hahaha woopsie!

 

Its hilarious when something silly like this can take down a company instead of a Denial of Service attack 

[spartaman64]

thats why there were no live channels that i follow on twitch for a time

[Sauron]

Come on... this is negligent at best...

 

You can't run such a huge cluster with a command line interface where typos in commands can have such drastic effects. You wouldn't manage a cloud storage service with freaking dd.

[Guest Kloaked]

9 hours ago, Misanthrope said:

 

This is a clickbaity post meant to be taken with humor. 

[Mr.Meerkat]

Oh lel, and that's why Grammar Nazis should exist in workplaces  

/s

 

15 hours ago, zMeul said:

- are you sure you want to format drive c: ? [Yes/No]

- [yes]

- formatting 100%

- wait .. what?!

If you're formatting the C drive (assuming it's the OS drive), wouldn't the OS be unable to continue formatting once past a certain point as it no longer exists? 

[DrMikeNZ]

15 hours ago, zMeul said:

- are you sure you want to format drive c: ? [Yes/No]

- [yes]

- formatting 100%

- wait .. what?!

Pixar. rm*

[Misanthrope]

2 hours ago, Sauron said:

Come on... this is negligent at best...

 

You can't run such a huge cluster with a command line interface where typos in commands can have such drastic effects. You wouldn't manage a cloud storage service with freaking dd.

For the most part I don't believe they actually type commands but rather create a bash script, test it, approve it and then use that. He probably invoked the wrong bash script or they missed something within the script itself. But well just speculation on my part.

[Stefan1024]

When we had risk management courses we saw some nice statistics. The chance for a failure to press a clearly labled butten by a well unstruced empolyee (a pretty simple task) is about 1/10 000. Now guess the chance for managing a data center. Mistakes just happen and you can't totally prevent, Only redundancy helps but only if the two systems are fully decoupled.

[AlexGoesHigh]

1 hour ago, Mr.Meerkat said:

Oh lel, and that's why Grammar Nazis should exist in workplaces  

/s

 

If you're formatting the C drive (assuming it's the OS drive), wouldn't the OS be unable to continue formatting once past a certain point as it no longer exists? 

It would still format because all the necessary components are in RAM, it wouldn't have to load anything from the drive to run.

 

I am talking of course at the OS level, if it was a high level program it would trow errors. Also if it wasn't possible to format the OS, then deleting system 32 manually shouldn't also be possible but well its still a prank trow around in multiplayer games.