Cloudflare DNS outage

[piratemonkey]

At about 2:15 PM (pacific time) Cloudflare experienced an outage for it's DNS services for about 25 minutes. Affected sites and services include Discord, Shopify, League of Legends, and many more. Google seems to have been affected (youtube and google search engine). Cloudflare's 1.1.1.1 seems to have gone down too

The problem seems to be fixed now

 

Quote

Many major websites and services were unreachable for a period Friday afternoon due to issues at Cloudflare’s 1.1.1.1 DNS service. The outage seems to have started at about 2:15 Pacific time and lasted for about 25 minutes before connections began to be restored. Google DNS may also have been affected.

Cloudflare at 2:46 says “the issue has been identified and a fix is being implemented.”

Discord, Feedly, Politico, Shopify and League of Legends were all affected, giving an idea of the breadth of the issue. Not only were websites down but also some status pages meant to provide warnings and track outages

 

I think this could've been a lot worse. With 1.1.1.1 being affected, people using it would not have been able to use the internet. Smh

Google seems to have been affected, this could be a bigger problem with internet companies (see twitter hack). 2020 is just a bad year

 

Sources

https://techcrunch.com/2020/07/17/cloudflare-dns-goes-down-taking-a-large-piece-of-the-internet-with-it/

People in comments (for 1.1.1.1 and google DNS)

Edited July 17, 2020 by piratemonkey
updated it for 1.1.1.1 and google being affected

[TempestCatto]

That's what happens when the internet runs off of like, two damn services. At least this happens very seldom though.

[Jumballi]

was using 1.1.1.1, took me down with it

 

edit: I should mention that all my devices are back up after I changed my dns settings

[piratemonkey]

Just now, Jumballi said:

was using 1.1.1.1, took me down with it

 

There goes the neighbourhood

[RAM555789]

I'm pretty sure Googles DNS took a slight hit too, on down detecor status, google, amazon web based service, and cloud flare all had issues, resulting in just about everything messing up.

[Helpful Tech Witch]

google did take a hit. youtube didnt work, neather did a google search

[piratemonkey]

6 minutes ago, RAM555789 said:

I'm pretty sure Googles DNS took a slight hit too, on down detecor status, google, amazon web based service, and cloud flare all had issues, resulting in just about everything messing up.

 

Just now, TheTechWizardThatNeedsHelp said:

google did take a hit. youtube didnt work, neather did a google search

thank you. will update

[SKHSVideo]

When it came back uBlock in FireFox stopped working for me... I forgot how many ads are on the internet... I also couldn't get to the options of uBlock.

 

Not knowing I did a reinstall of FireFox.

[piratemonkey]

8 minutes ago, SKHSVideo said:

When it came back uBlock in FireFox stopped working for me... I forgot how many ads are on the internet... I also couldn't get to the options of uBlock.

 

Not knowing I did a reinstall of FireFox.

That's interesting. I wasn't using FireFox during the outage, but I do use uBlock Origin on Chrome. I didn't notice anything out of the norm (though I don't aggressively block like FireFox, only for trackers)

[lewdicrous]

This sums up everyone's reaction.

This isn't the first time something like this happens. We need a backup in case it happens again and is out for longer.

[piratemonkey]

4 minutes ago, lewdicrous said:

This sums up everyone's reaction.

*snip*

This isn't the first time something like this happens. We need a backup in case it happens again and is out for longer.

That's accurate

[justpoet]

I thought I was actually pretty well isolated from occurrences like this.  I run my own DNS caching server, it does forward lookups through 1.1.1.2 normally, but also has backups through 4.4.4.4, 9.9.9.9, and a couple ISPs.  I still couldn't get anything to be looked up unless it was very recent in the cache, even though I could ping 1.1.1.2 and 9.9.9.9 just fine.  During that time, FB stopped loading for me too, even though I had all of their DNS info cached.

 

This feels more like a takedown of a part of the backbone, which happened to also take out DNS servers for some CDNs, rather than just DNS issues.  I look forward to hearing the full on security analysis of it all once that's done.

[justpoet]

OK, looks like the basic info is actually out already.

 

Quote

Update - This afternoon we saw an outage across some parts of our network. It was not as a result of an attack. It appears a router on our global backbone announced bad routes and caused some portions of the network to not be available. We believe we have addressed the root cause and are monitoring systems for stability now.

...

...

...

Data Centers impacted include: SJC, DFW, SEA, LAX, ORD, IAD, EWR, ATL, LHR, AMS, FRA, CDG

Quote

With regards to the recent outage that we experienced with our DNS resolver and network, our team has identified that this issue appears to be related to our routing backbone that caused bad routes to be announced which caused some portions of the network to be unavailable.

...

The time stamp for Incident is between 21:11 and 22:05 UTC;

 

So, yep...

 

Above taken from various Cloudflare responses on their community forums.  Official incident update here:

https://www.cloudflarestatus.com/incidents/b888fyhbygb8

[piratemonkey]

Just now, justpoet said:

OK, looks like the basic info is actually out already.

From where? Cloudflare's support page?

[justpoet]

1 minute ago, piratemonkey said:

From where? Cloudflare's support page?

Yep.  They say it wasn't an attack, but that's a weird thing to have suddenly start happening.

[jasonvp]

35 minutes ago, justpoet said:

Yep.  They say it wasn't an attack, but that's a weird thing to have suddenly start happening.

It likely didn't "suddenly start happening" for any reason other than maintenance.  Someone probably goofed up and pushed a maintenance when he/she shouldn't have.  Note:

 

We believe we have addressed the root cause

 

Which can be read as: "We found the the mistake the person made and un-did it."

[Kisai]

3 hours ago, Jumballi said:

was using 1.1.1.1, took me down with it

 

edit: I should mention that all my devices are back up after I changed my dns settings

That's why you have more than one DNS.

 

Personally I just leave the DNS set to the ISP, and 8.8.8.8 (google) is the secondary, not the primary.

[RejZoR]

There's also NextDNS

[RejZoR]

6 minutes ago, huilun02 said:

I use OpenDNS...

 

NextDNS is nice but its going paid. Didn't use it because it wouldn't take a self made custom domain block list.

NextDNS is still free. Just not unconditionally like in BETA. It's fully free if you only use it for domain resolving. If you use blocklists, you get 150.000 queries/month for free. Once you get past that it'll operate as regular DNS without any blocklists till next month begins and then it starts over. It's like 20€/year if you go unlimited.

 

I get through month with the free plan just fine and really don't see a reason why I'd need a custom block list given how many they offer and you can also add your own blacklist entries. I've needed to block some stuff on top of list and I've done it through blacklist (now Denylist coz racism is a real thing in software LMAO XD).

[piratemonkey]

2 minutes ago, huilun02 said:

my self made block list is almost 3,000 lines of entries that organisations really dont want people blocking

Like what?

[piratemonkey]

7 minutes ago, huilun02 said:

Mostly telemetry, data collection and tracking servers of the big players like Google, Microsoft. And manufacturer of my equipment that are also using the DNS service (for example blocking Samsung, Oneplus, and Xiaomi's naughty stuff because I use those brands of devices) And some ads/tracking delivery networks that are missed out.

 

Entries all of which I could not find in the available selectable block lists so I had to compile them into my own.

Good job. I'm just using uBlock origin with several lists. I need to up my blocking game

[TehDwonz]

GRC's DNSBench is a good way to find the lowest latency DNS to use, based on your location in cyberspace. Try to pick 3-4 different "providers" for your list.

 

Here's mine, set on the firewall so everything uses them:

 

DNSBench is here: https://www.grc.com/dns/benchmark.htm and this tool will also tell you if the provider redirects non-existent domains to adverts etc.

 

[RejZoR]

3 hours ago, huilun02 said:

Mostly telemetry, data collection and tracking servers of the big players like Google, Microsoft. And manufacturer of my equipment that are also using the DNS service (for example blocking Samsung, Oneplus, and Xiaomi's naughty stuff because I use those brands of devices) And some ads/tracking delivery networks that are missed out.

 

Entries all of which I could not find in the available selectable block lists so I had to compile them into my own.

For which NextDNS provides OS level lists. There is also NoGoogle list that blocks everything Google. Also LightSwitch05 blocks telemetry from MS and bunch of others. If you need over 600.000 queries a month, then 20€ a year is not exactly expensive given you can then filter unlimited number of devices and queries...

[Doobeedoo]

Internet is down, time to log off everyone... 

[jasonvp]

15 hours ago, jasonvp said:

Which can be read as: "We found the the mistake the person made and un-did it."

Looks like CloudFlare published an RCA of sorts which basically backs up what I suspected: someone goofed.

 

Quote

The outage occurred because, while working on an unrelated issue with a segment of the backbone from Newark to Chicago, our network engineering team updated the configuration on a router in Atlanta to alleviate congestion. This configuration contained an error that caused all traffic across our backbone to be sent to Atlanta. This quickly overwhelmed the Atlanta router and caused Cloudflare network locations connected to the backbone to fail.

 

 

Maintenance on a Friday afternoon.  Never, ever a good idea.  Heh.  What could possibly go wrong?!