Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
Sign in to follow this  
piratemonkey

Cloudflare DNS outage

Recommended Posts

3 hours ago, huilun02 said:

Mostly telemetry, data collection and tracking servers of the big players like Google, Microsoft. And manufacturer of my equipment that are also using the DNS service (for example blocking Samsung, Oneplus, and Xiaomi's naughty stuff because I use those brands of devices) And some ads/tracking delivery networks that are missed out.

 

Entries all of which I could not find in the available selectable block lists so I had to compile them into my own.

For which NextDNS provides OS level lists. There is also NoGoogle list that blocks everything Google. Also LightSwitch05 blocks telemetry from MS and bunch of others. If you need over 600.000 queries a month, then 20€ a year is not exactly expensive given you can then filter unlimited number of devices and queries...

Link to post
Share on other sites
3 hours ago, RejZoR said:

 

Maybe if you read the last sentence in my previous post...


Awareness is key. Never enough, even in the face of futility. Speak the truth as if you may never get to say it again. This world is full of ugly. Change it they say. The only way is to reveal the ugly. To change the truth you must first acknowledge it. Never pretend it isn't there. Never bend the knee.

 

Please quote my post in your reply, so that I will be notified and can respond to it. Thanks.

Link to post
Share on other sites

Internet is down, time to log off everyone... 


Ryzen 7 3800X | X570 Aorus Elite | G.Skill 16GB 3200MHz C16 | Radeon RX 5700 XT | Samsung 850 PRO 256GB | Mouse: Zowie S1 | OS: Windows 10

Link to post
Share on other sites
15 hours ago, jasonvp said:

Which can be read as: "We found the the mistake the person made and un-did it."

Looks like CloudFlare published an RCA of sorts which basically backs up what I suspected: someone goofed.

 

Quote

The outage occurred because, while working on an unrelated issue with a segment of the backbone from Newark to Chicago, our network engineering team updated the configuration on a router in Atlanta to alleviate congestion. This configuration contained an error that caused all traffic across our backbone to be sent to Atlanta. This quickly overwhelmed the Atlanta router and caused Cloudflare network locations connected to the backbone to fail.

 

 

Maintenance on a Friday afternoon.  Never, ever a good idea.  Heh.  What could possibly go wrong?!

 


Editing Rig: Mac Pro 7,1

System Specs: 3.2GHz 16-core Xeon | 96GB ECC DDR4 | AMD Radeon Pro Vega II (32GB HBM2) | Lots of SSD and NVMe storage |

Audio: Sound Blaster X7 external DAC/ADC |

 

Gaming Rig: PC

System Specs:  Asus Rampage VI Extreme board | Intel Core i9 7900X | 64GB Corsair Vengeance LPX (OC'd to 4GHz) | 2 x NVidia 2080Ti FE cards (OC'd) | Corsair AX1500i power supply | CaseLabs Magnum THW10 case (RIP CaseLabs ) |

Audio:  Sound Blaster AE-9 card | Mackie DL32R Mixer | Sennheiser HDV820 amp | Sennheiser HD820 phones | Rode Broadcaster mic |

Displays: Asus PG27UQ 4K/144Hz display | 2 x LG 27UK650-W 4K displays |

Cooling:  2 x EK 140 Revo D5 Pump/Res | EK Asus R6E monoblock | 2 x EK 2080Ti waterblocks | 2 x AlphaCool 480mm x 60mm rads | AlphaCool 560mm x 60mm rad | 21 x Noctua 120mm fans | 8 x Noctua 140mm fans | 2 x Aquaero 6XT fan controllers |

Link to post
Share on other sites
7 hours ago, TehDwonz said:

GRC's DNSBench is a good way to find the lowest latency DNS to use, based on your location in cyberspace. Try to pick 3-4 different "providers" for your list.

 

Here's mine, set on the firewall so everything uses them:
image.png.f211264cbaaa58f69eb0d1151a58f43f.png

 

DNSBench is here: https://www.grc.com/dns/benchmark.htm and this tool will also tell you if the provider redirects non-existent domains to adverts etc.

 

Dns latency for most is rather pointless. It only affects the time it takes for the name to Ip translation to happen. So even if we are talking about a 2000 ms ping we are talking 2 seconds. So it isn't even a big inconvenience in that case. 

 

I mean if you KNEW to look for it you might be able to notice a 30ms vs 2000ms delay from dns, but I think the vast majority wouldn't even know it was slow. Plus 2000 ms isn't even a real scenario just an example I am using.

Link to post
Share on other sites
10 minutes ago, AngryBeaver said:

Dns latency for most is rather pointless. It only affects the time it takes for the name to Ip translation to happen. So even if we are talking about a 2000 ms ping we are talking 2 seconds. So it isn't even a big inconvenience in that case. 

 

I mean if you KNEW to look for it you might be able to notice a 30ms vs 2000ms delay from dns, but I think the vast majority wouldn't even know it was slow. Plus 2000 ms isn't even a real scenario just an example I am using.

 

You can't tell the difference between 30ms and 2 seconds? Really?

 

Did you see the part about redirects being flagged too? :) 

Link to post
Share on other sites
3 hours ago, TehDwonz said:

 

You can't tell the difference between 30ms and 2 seconds? Really?

 

Did you see the part about redirects being flagged too? :) 

My point is that it isn't super obvious because of how fast internet speeds are and once the connection is established dns delay isn't part of the equation.

 

https://www.dnsperf.com/

 

So the fastest average is 12ms

the slowest average is 132ms

 

so yes one is 11 times higher that the other... but we are talking about a 100th of a second compared to a 10th of a second. That, IMO, much quicker than most people would ever notice.

 

Now if the DNS speed is super slow (in the range of seconds) it is possible to have a webpage load and have ads and pictures that might be coming from other domains take that delay to show up... which can make things pop in or cause the page to load images and scroll on you, but again we aren't talking about a huge amount of time here.

Link to post
Share on other sites
2 minutes ago, AngryBeaver said:

My point is that it isn't super obvious because of how fast internet speeds are and once the connection is established dns delay isn't part of the equation.

It was never really the issue, more a point of interest. It was about using multiple providers. I posted a way to find some reliable ones and/or geo-local ones, and also to flag any that redirect to ads if you make a typo and try to access a non-existent domain. In addition, the tool tests DNSSEC authentication. So for anyone wanting their own list of DNS providers to use, it's a good tool.

Link to post
Share on other sites
1 hour ago, TehDwonz said:

It was never really the issue, more a point of interest. It was about using multiple providers. I posted a way to find some reliable ones and/or geo-local ones, and also to flag any that redirect to ads if you make a typo and try to access a non-existent domain. In addition, the tool tests DNSSEC authentication. So for anyone wanting their own list of DNS providers to use, it's a good tool.

Pi hole with DOT or DOH for dns sec and filtering.  If you are trying to just protect your family for free then 

 

Filtering out malicious sites

1.1.1.2

1.0.0.2   

 

For filtering out adult content and malicious sites.

1.1.1.3

1.0.0.3

Link to post
Share on other sites
On 7/18/2020 at 11:41 AM, jasonvp said:

Looks like CloudFlare published an RCA of sorts which basically backs up what I suspected: someone goofed.

 

 

Maintenance on a Friday afternoon.  Never, ever a good idea.  Heh.  What could possibly go wrong?!

 

Yeah, no kidding! 😡

 

Our local data center NOC thought it was a swell idea to perform UPS maintenance and generator testing without informing tenants (that was later corrected). What happened was when they performed a test, it caused a bank of UPS units to fail. This shifted the load to the other leg and thus a cascade failure when the entire data center was without power.

 

Dirty shutdowns on a SAN full of running VMs is never good. A few of them were so corrupted that I had to restore from backup. We spend a good portion the night getting the effected networks back up and running. I never did go home that night to get sleep. I literally worked 2x 8 hour shifts back to back.

 

And yeah, I'm still pissed about that!

 

This kind of failure should never happen. To make such sweeping changes of this magnitude should involve a team of engineers to cross the "T"s and dot the "I"s. Check, double-check, triple-check, sign off on final deliverable of change by management.

Link to post
Share on other sites

I had this, so I didn't notice. I mean my internet alarm (my kid) didnt notice. :D  

 

Static DNS 1 1.1.1.1
Static DNS 2 1.0.0.1
Static DNS 3 9.9.9.9

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  


×