outage Facebook, WhatsApp and Instagram face outage[Fixed now]

[Arika]

1 hour ago, Master Delta Chief said:

What do you expect? social media is one of the worst social media platforms out there for a whole lot of reasons. 

Fixed.

there is no "worst" social media, all social media platforms are as shit as each other.

[Fasterthannothing]

6 hours ago, Mihle said:

Based on this:

I havent found any article that backs it up, but a theory I have is that problems with home internet at the same time as this is because of this. That the Internet Providers DNS resolver got overloaded.

At least, the problems was worked for few sec but slow then didn't at all for a while and repeat, something that could be a sign of overloading.

 

Just a theory tho. Didn't think of trying to change DNS resolver, but didn't feel I needed to fix it yesterday either so...

Yeah my internet never blipped because I run my own resolver

[Lord Szechenyi]

7 hours ago, Salv8 (sam) said:

we achieved a utopia for for 6 hrs...

WHY DID IT HAVE TAKEN AWAY?!?!?!?!?

is that English?

 

[hngaminguk]

Not sure if this has been shared already but https://blog.cloudflare.com/october-2021-facebook-outage/

 

Technically it wasn't a DNS issue but actually BGP, it's very interesting that stuff like this can even happen in our day and age. We serious need an improvement on this and this is why I think people need to be looking at Syntropy more!

 

 

[igormp]

3 minutes ago, hngaminguk said:

think people need to be looking at Syntropy more!

Are you memeing or are you serious? If the latter, then this seems to be built on top of our current network protocols, so it would have died in the same way in case of a BGP failure.

[hngaminguk]

11 minutes ago, igormp said:

Are you memeing or are you serious? If the latter, then this seems to be built on top of our current network protocols, so it would have died in the same way in case of a BGP failure.

I am serious, Syntropy is designed from the ground up to be smarter than current routing algorithms, security features can be built on top of it. eg Preventing events such as BGP hijacking. 

 

There is a lot more info on their site and a sizeable community on Telegram that can provide information in a better way than I can here. 

 

We obviously don't have the full details on how this happened and hopefully Facebook will provide a detailed post mortem (don't count on it we know what they are like) Because of the security features that can be added with Syntropy this potentially could have been a small blip or even non issue.

[igormp]

1 minute ago, hngaminguk said:

Preventing events such as BGP hijacking. 

It wasn't a hijack in this case tho.

 

4 minutes ago, hngaminguk said:

We obviously don't have the full details on how this happened and hopefully Facebook will provide a detailed post mortem (don't count on it we know what they are like)

They have a really shallow one that doesn't go into details: https://engineering.fb.com/2021/10/04/networking-traffic/outage/

I believe they'll share a more detailed one later on.

5 minutes ago, hngaminguk said:

Because of the security features that can be added with Syntropy this potentially could have been a small blip or even non issue.

From what I saw at their website, at best we would've just known that their BGP routes were gone (which Cloudflare already did), but not why, nor have a way to avoid that.

[hngaminguk]

3 minutes ago, igormp said:

It wasn't a hijack in this case tho.

 

They have a really shallow one that doesn't go into details: https://engineering.fb.com/2021/10/04/networking-traffic/outage/

I believe they'll share a more detailed one later on.

From what I saw at their website, at best we would've just known that their BGP routes were gone (which Cloudflare already did), but not why, nor have a way to avoid that.

Not saying that it was a hijack in this case, just that it is something that Syntropy could protect against. 

 

In relation to " at best we would've just known that their BGP routes were gone" have a look at the following (more detailed) post around how DARP works: https://www.linkedin.com/pulse/distributed-autonomous-routing-protocol-darp-william-b-norton

[igormp]

7 minutes ago, hngaminguk said:

In relation to " at best we would've just known that their BGP routes were gone" have a look at the following (more detailed) post around how DARP works: https://www.linkedin.com/pulse/distributed-autonomous-routing-protocol-darp-william-b-norton

It's interesting and utopic, but still wouldn't have changed anything here. It'd have been as if a node simply went missing, so the issue would still be the same.

[Helpful Tech Witch]

10 hours ago, mr moose said:

Along with twitter, reddit, whatsapp, instagram...   Basically most of what the internet is used for.

 

Hell even imgur is hell hole full of petulant ignorance at times.

What?

 WhatsApp is texting.

 It’s that really that much of a bad thing?

[hngaminguk]

1 minute ago, igormp said:

It's interesting and utopic, but still wouldn't have changed anything here. It'd have been as if a node simply went missing, so the issue would still be the same.

Potentially yes. 

 

This is also where decentralisation would come into play a lot here! If Facebook were properly decentralised and had systems running on multiple different providers (not just relying on them being their own provider) then some systems may have died but not all.

 

At my place of work if we have a full outage of any longer than 1 - 2 hours in a year we would have to provide refund credits to the customer. It boggles the mind to how an issue like this even happened given the size of the company. You'd think/hope there are procedures in place to ensure something like this has the smallest chance of happening. 

 

Again until Facebook provide a full and detailed report on what happened we shall never fully know. 

[Kilrah]

4 hours ago, HelpfulTechWizard said:

What?

 WhatsApp is texting.

 It’s that really that much of a bad thing?

In itself no, but being tied into other FB stuff is where the questionability is. 

[Master Delta Chief]

Facebook corp should be broken up with Instagram and Whatsapp becoming independent once again considering Facebook has become a monopoly. 

[mr moose]

17 hours ago, HelpfulTechWizard said:

What?

 WhatsApp is texting.

 It’s that really that much of a bad thing?

Don't even use it, but as I said:

On 10/5/2021 at 6:10 PM, mr moose said:

I was just reeling therm off the top of my head,

 

 

Having a think about though,  it's not so much what the app does, but what the company does with the data that makes it as much a cancer as being a social media would.

[Beskamir]

18 hours ago, mr moose said:

I was just reeling therm off the top of my head, I'd also add linkedin to that list,  It's not supposed to be social media but it's a cluster fuck of pseudo professionals who all seem to to use it as a dick measuring tool.

Yeah Linkedin is easy to forget cause I rarely get reminded of it.

[igormp]

On 10/5/2021 at 11:36 AM, hngaminguk said:

Again until Facebook provide a full and detailed report on what happened we shall never fully know. 

They provided a new one, but with not so much technical detail:

https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/

[JogerJ]

Pretty much this video by Computerphile summarized it best in layman terms. So in the end it wasn't BGP hijacking, but for some reason Facebook stopped broadcasting their address to other AS and they went poof from the internet. The facebook servers are most likely still connected to the internet but it's just no one could find where facebook.com is, so all we know was facebook.com didn't exist in the network!

 

Why Did Facebook Go Down? - Computerphile - YouTube

 

Also if you want to look around with the visualization of it as it happens: RIPEstat - BGPlay

 

[hngaminguk]

6 hours ago, igormp said:

They provided a new one, but with not so much technical detail:

https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/

That is more information than I expected from Facebook to be honest!

 

Notable info from that:

 

"During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally. Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool prevented it from properly stopping the command. " 

 

"they faced two large obstacles: first, it was not possible to access our data centers through our normal means because their networks were down, and second, the total loss of DNS broke many of the internal tools we’d normally use to investigate and resolve outages like this. "

 

The second quote I have there is just... wtf 

 

They mean to say they don't have completely separate systems for remote management... I get that the core networking had issues due to their miss configuration but they should at minimum have either completely separate systems for management OR engineers on site in case of an issue like this.