Jump to content

Facebook, WhatsApp and Instagram face outage[Fixed now]

[The issue has been resolved as of 21:30 UTC. It lasted for about 6 hours.]

Summary

Facebook, WhatsApp and Instagram, Facebook Messenger and Oculus VR are facing outages since the past half an hour(15:40 UTC)

 

Quotes

   The Verge:

Quote

Facebook's network of services suddenly went offline. On Twitter, Facebook communications exec Andy Stone says:

Spoiler

A peek at Down Detector reveals the problems are widespread. While it’s unclear exactly why they are unreachable for so many people, but a peek at their DNS records shows that, the problem is apparently DNS (it’s always DNS).
Instagram.com is flashing a 5xx Server Error message, while the Facebook site merely tells us that something went wrong. The problem also appears to be affecting its virtual reality arm, Oculus. Users can load games they already have installed and the browser works, but social features or installing new games does not.

   Independent.co.uk:

Quote

The three apps – which are all owned by Facebook, and run on shared infrastructure – all completely stopped working shortly before 5pm. Other products that are part of the same family of apps, such as Facebook Workplace, also stopped working.

Visitors to the Facebook website simply saw an error page or a message that their browser could not connect. The WhatsApp and Instagram apps continued to work, but did not show new content, including any messages sent or received during the problems.

 

UPDATES

Spoiler

A bit more information on the issue. Thanks to @tikker.

Quote

This was a reddit comment from a user who claims to be a Facebook employee working on the issue. The comment was apparently deleted.

As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).

There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.

Part of this is also due to lower staffing in data centers due to pandemic measures.

Spoiler

 

 

 

Spoiler

This reddit comment was from a Facebook employee who claims to be working on the issue. This comment too was deleted.

No discussion that I'm aware of yet that is considering a threat/attack vector.

I believe the original change was 'automatic' (as in configuration done via a web interface). However, now that connection to the outside world is down, remote access to those tools don't exist anymore, so the emergency procedure is to gain physical access to the peering routers and do all the configuration locally.

Spoiler

 

 

Spoiler

The issue is resolved as of 21:30 UTC.

Quote from The Verge:

Quote

After failing all tests for most of the day, a test of ISP DNS servers via DNSchecker.org showed most of them successfully finding a route to Facebook.com at 5:30PM ET. A few minutes later, we were able to start using Facebook and Instagram normally; however, it may take time for the DNS fixes to reach everyone.

 

 

THE ISSUE

An article from Cloudflare explains the issue that occurred well(Thanks @igormp):

Spoiler

Info: The Internet is literally a network of networks, and it’s bound together by BGP. BGP allows one network (say Facebook) to advertise its presence to other networks that form the Internet. As we write Facebook is not advertising its presence, ISPs and other networks can’t find Facebook’s network and so it is unavailable.

The individual networks each have an ASN: an Autonomous System Number. An Autonomous System (AS) is an individual network with a unified internal routing policy. An AS can originate prefixes (say that they control a group of IP addresses), as well as transit prefixes (say they know how to reach specific groups of IP addresses).

 

- At 1658 UTC we noticed that Facebook had stopped announcing the routes to their DNS prefixes. That meant that, at least, Facebook’s DNS servers were unavailable. Because of this Cloudflare’s 1.1.1.1 DNS resolver could no longer respond to queries asking for the IP address of facebook.com or instagram.com.

- Meanwhile, other Facebook IP addresses remained routed but weren’t particularly useful since without DNS Facebook and related services were effectively unavailable.

But at around 15:40 UTC we saw a peak of routing changes from Facebook. That’s when the trouble began.

- As a direct consequence of this, DNS resolvers all over the world stopped resolving their domain names.

- Due to Facebook stopping announcing their DNS prefix routes through BGP, our and everyone else's DNS resolvers had no way to connect to their nameservers. Consequently, 1.1.1.1, 8.8.8.8, and other major public DNS resolvers started issuing (and caching) SERVFAIL responses.

 

My thoughts

I think this is gonna cost them a lot of money. The outage has been continuing for the past 40 minutes at this point. While I don't use Facebook and Instagram, I'm sure many do and aren't too happy about that as well. WhatsApp, my main convo app not working is pretty bad, and it is also the main conversation app for a lot of people.

 

Sources

https://www.theverge.com/2021/10/4/22708989/instagram-facebook-outage-messenger-whatsapp-error

https://www.independent.co.uk/life-style/gadgets-and-tech/facebook-down-instagram-whatsapp-today-b1932059.html

https://blog.cloudflare.com/october-2021-facebook-outage/

 

Errors that can be seen

Spoiler

WhatsApp:

Spoiler

IMG_3067-1.jpg.c284e34e55442d300d69876d44796716.jpg

 

Facebook:

Spoiler

image.thumb.png.c94c856c6b77d2c4f5624aa882229d04.png

 

Instagram:

Spoiler

image.png.e7775b84c5861bc2c1fc913cd07b196a.png

 

 

Edited by RockSolid1106
Minor changes to formatting.

"Coding skill isn’t about knowing how to make things; It’s about being able to find the right Stack Overflow code to copy paste."

 

bruh switch to dark mode its at the bottom of this page

VPN Server Guide

Link to comment
Share on other sites

Link to post
Share on other sites

Good [temporary] riddance... - though it does make me glad that SMS is still reasonably well-known

____________________________________________________________________________________________________________________________________

 

 

____________________________________________________________________________________________________________________________________

pythonmegapixel

into tech, public transport and architecture // amateur programmer // youtuber // beginner photographer

Thanks for reading all this by the way!

By the way, my desktop is a docked laptop. Get over it, No seriously, I have an exterrnal monitor, keyboard, mouse, headset, ethernet and cooling fans all connected. Using it feels no different to a desktop, it works for several hours if the power goes out, and disconnecting just a few cables gives me something I can take on the go. There's enough power for all games I play and it even copes with basic (and some not-so-basic) video editing. Give it a go - you might just love it.

Link to comment
Share on other sites

Link to post
Share on other sites

Instagram has one or two of these a year. Error 5xx. So it's not a uncommon occurence. I'm not sure about facebook and whatsapp, but i believe they are on the same servers as instagram. 

https://www.lttstore.com/

1990 M3s are the best looking things ever made.    

^This statement has been retracted^
2020/2021 BMW S1000RRs/Ninja H2s are the best looking things ever made. 

Don't ask to ask. 
If you want me to see the reply, @XGoodGuyFitz(aka me) and/or quote me.
Thanks!

 

Link to comment
Share on other sites

Link to post
Share on other sites

1 minute ago, XGoodGuyFitz said:

Instagram has one or two of these a year. Error 5xx. So it's not a uncommon occurence. I'm not sure about facebook and whatsapp, but i believe they are on the same servers as instagram. 

In this case it could be something major, even the DNS servers are gone....

Link to comment
Share on other sites

Link to post
Share on other sites

Just now, jagdtigger said:

In this case it could be something major, even the DNS servers are gone....

Well, that kinda sucks. They better get it back online cuz thats how i talk to people after school. 

https://www.lttstore.com/

1990 M3s are the best looking things ever made.    

^This statement has been retracted^
2020/2021 BMW S1000RRs/Ninja H2s are the best looking things ever made. 

Don't ask to ask. 
If you want me to see the reply, @XGoodGuyFitz(aka me) and/or quote me.
Thanks!

 

Link to comment
Share on other sites

Link to post
Share on other sites

2 minutes ago, jagdtigger said:

In this case it could be something major, even the DNS servers are gone....

OH i almost forgot, if it is not fixed within 12 hours, it is definitely something major as the 5xx error doesn't take more than 12 hours to fix. 

https://www.lttstore.com/

1990 M3s are the best looking things ever made.    

^This statement has been retracted^
2020/2021 BMW S1000RRs/Ninja H2s are the best looking things ever made. 

Don't ask to ask. 
If you want me to see the reply, @XGoodGuyFitz(aka me) and/or quote me.
Thanks!

 

Link to comment
Share on other sites

Link to post
Share on other sites

Just now, jagdtigger said:

Meh, it wouldnt be a huge loss if it doesnt came back....

that would be a huge loss. 

https://www.lttstore.com/

1990 M3s are the best looking things ever made.    

^This statement has been retracted^
2020/2021 BMW S1000RRs/Ninja H2s are the best looking things ever made. 

Don't ask to ask. 
If you want me to see the reply, @XGoodGuyFitz(aka me) and/or quote me.
Thanks!

 

Link to comment
Share on other sites

Link to post
Share on other sites

4 minutes ago, jagdtigger said:

Meh, it wouldnt be a huge loss if it doesnt come back....

Got my English test tomorrow, what do you mean! Most of the communication is on WhatsApp!

I mean I'll be happy if it gets canceled xD

"Coding skill isn’t about knowing how to make things; It’s about being able to find the right Stack Overflow code to copy paste."

 

bruh switch to dark mode its at the bottom of this page

VPN Server Guide

Link to comment
Share on other sites

Link to post
Share on other sites

11 minutes ago, XGoodGuyFitz said:

Well, that kinda sucks. They better get it back online cuz thats how i talk to people after school. 

5 minutes ago, XGoodGuyFitz said:

that would be a huge loss. 

Not really. People would just migrate to Signal, and then the world would be a much better place without Facebook's dubious data mining.

 

The greatest loss would probably be the huge amounts of data that people have invested in these platforms - think holiday photos,  that kind of thing. But frankly anyone who puts their data into a free online service and maintains no other backups will almost certainly get zero sympathy from me if they lose it.

____________________________________________________________________________________________________________________________________

 

 

____________________________________________________________________________________________________________________________________

pythonmegapixel

into tech, public transport and architecture // amateur programmer // youtuber // beginner photographer

Thanks for reading all this by the way!

By the way, my desktop is a docked laptop. Get over it, No seriously, I have an exterrnal monitor, keyboard, mouse, headset, ethernet and cooling fans all connected. Using it feels no different to a desktop, it works for several hours if the power goes out, and disconnecting just a few cables gives me something I can take on the go. There's enough power for all games I play and it even copes with basic (and some not-so-basic) video editing. Give it a go - you might just love it.

Link to comment
Share on other sites

Link to post
Share on other sites

1 minute ago, pythonmegapixel said:

Not really. People would just migrate to Signal, and then the world would be a much better place without Facebook's dubious data mining.

 

The greatest loss would probably be the huge amounts of data that people have invested in these platforms - think holiday photos,  that kind of thing. But frankly anyone who puts their data into a free online service and maintains no other backups will almost certainly get zero sympathy from me if they lose it.

is signal another kind of social media or sumn?

https://www.lttstore.com/

1990 M3s are the best looking things ever made.    

^This statement has been retracted^
2020/2021 BMW S1000RRs/Ninja H2s are the best looking things ever made. 

Don't ask to ask. 
If you want me to see the reply, @XGoodGuyFitz(aka me) and/or quote me.
Thanks!

 

Link to comment
Share on other sites

Link to post
Share on other sites

1 minute ago, XGoodGuyFitz said:

is signal another kind of social media or sumn?

It's an open-source Whatsapp equivalent maintained by a non-profit group specifically dedicated to it, rather than by Facebook. I say equivalent because IIRC it actually implements the same protocol as Whatsapp so it's still private/secure.

 

It works really well, but I never use it because I can't persuade anyone else to switch!

____________________________________________________________________________________________________________________________________

 

 

____________________________________________________________________________________________________________________________________

pythonmegapixel

into tech, public transport and architecture // amateur programmer // youtuber // beginner photographer

Thanks for reading all this by the way!

By the way, my desktop is a docked laptop. Get over it, No seriously, I have an exterrnal monitor, keyboard, mouse, headset, ethernet and cooling fans all connected. Using it feels no different to a desktop, it works for several hours if the power goes out, and disconnecting just a few cables gives me something I can take on the go. There's enough power for all games I play and it even copes with basic (and some not-so-basic) video editing. Give it a go - you might just love it.

Link to comment
Share on other sites

Link to post
Share on other sites

2 minutes ago, Latvian Video said:

And many other sites are down, just check https://downdetector.com/

That's a "What sites are down?", "yes" situation if I've ever seen one lol. Since Cloudflare is on the list I wonder if that's one of the reasons.

Crystal: CPU: i7 7700K | Motherboard: Asus ROG Strix Z270F | RAM: GSkill 16 GB@3200MHz | GPU: Nvidia GTX 1080 Ti FE | Case: Corsair Crystal 570X (black) | PSU: EVGA Supernova G2 1000W | Monitor: Asus VG248QE 24"

Laptop: Dell XPS 13 9370 | CPU: i5 10510U | RAM: 16 GB

Server: CPU: i5 4690k | RAM: 16 GB | Case: Corsair Graphite 760T White | Storage: 19 TB

Link to comment
Share on other sites

Link to post
Share on other sites

14 minutes ago, jagdtigger said:

Meh, it wouldnt be a huge loss if it doesnt came back....

It would be

WhatssApp is the main messing app for a lot of people.

 

I could use some help with this!

please, pm me if you would like to contribute to my gpu bios database (includes overclocking bios, stock bios, and upgrades to gpus via modding)

Bios database

My beautiful, but not that powerful, main PC:

prior build:

Spoiler

 

 

Link to comment
Share on other sites

Link to post
Share on other sites

Some info:

Quote

This is a global outage for all FB-related services/infra (source: I'm currently on the recovery/investigation team).

Update 1440 UTC:

As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).

There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.

Part of this is also due to lower staffing in data centers due to pandemic measures.

 

 

 

Edited by GDRRiley
MOD: fixed text color

Crystal: CPU: i7 7700K | Motherboard: Asus ROG Strix Z270F | RAM: GSkill 16 GB@3200MHz | GPU: Nvidia GTX 1080 Ti FE | Case: Corsair Crystal 570X (black) | PSU: EVGA Supernova G2 1000W | Monitor: Asus VG248QE 24"

Laptop: Dell XPS 13 9370 | CPU: i5 10510U | RAM: 16 GB

Server: CPU: i5 4690k | RAM: 16 GB | Case: Corsair Graphite 760T White | Storage: 19 TB

Link to comment
Share on other sites

Link to post
Share on other sites

20 minutes ago, tikker said:

That's a "What sites are down?", "yes" situation if I've ever seen one lol. Since Cloudflare is on the list I wonder if that's one of the reasons.

I'm not sure how reliable this is. I've been using Amazon and Gmail all day without any problem.

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×