Route Leak causes Cloudflare to go down

[Tamesh16]

CloudFlare was down for about over an hour

https://www.cloudflarestatus.com/incidents/46z55mdhg0t5

 

"We have identified a possible route leak impacting some Cloudflare IP ranges"  - Jun 24, 11:36 UTC

 

"The network responsible for the route leak has now fixed the issue." - Jun 24, 12:42 UTC

 

Down detector suggests main reports coming from East coast of the US and Europe.

 

This has caused Discord to go down as well , the verge posted an update about this

https://www.theverge.com/2019/6/24/18715308/discord-down-outage-cloudflare-problems-crunchyroll-feedly

even though Cloudflare has posted that updates have been fixed discord remains down  * at the time of posting this update * 

 

"We are working on resolving some internal technical problems now." - Discord, Jun 24, 13:07 UTC

https://status.discordapp.com/

 

I had trouble accessing the forums for a little while, but it wasnt as much as Discord

 

EDIT : Issue marked as fully resolved by Cloudflare

"Traffic levels have returned to normal now that the route leak has been fixed. We are now marking this incident as resolved." -Jun 24, 13:02 UTC

 

EDIT: Discord Identified issues and working on fix

"The issue has been identified and a fix is being implemented." -Jun 24, 13:19 UTC

 

EDIT: Discord Update

"We've identified the issue and people should start reconnecting over time." -Jun 24, 13:37 UTC

 

EDIT: Discord should be back up, they are Monitoring the situation

"Users should all be able to reconnect now."-Jun 24, 14:06 UTC

 

EDIT: DIscord back up and problem fully resolved

"The issue is resolved, thank you for your patience! We will release a public post mortem for this one." Jun 24, 2019 - 14:38 UTC

This means discord was down for about an hour and half

 

This is my first news post, sorry if i left out anything, this is all i could find on the topic. If you have anything to Add, please do ! 

Hopefully you werent affected as much by this event

Edited June 24, 2019 by Tamesh16

[Princess Luna]

Yep, time to go back to Skype? no? oh well.

 

At least Telegram is working flawlessly.

[TVwazhere]

The outage temporarily caused parts of LTT to also not work for a brief period. Should all be good now though. 

[Tamesh16]

Just now, TVwazhere said:

The outage temporarily caused parts of LTT to also not work for a brief period. Should all be good now though. 

i was replying to a post when it went down and i couldnt get back on for a while, but yes it seems all good now 

 

[Princess Luna]

[robindb97]

sadly discord is still down

 

 

We are working on resolving some internal technical problems now. 

[Tamesh16]

16 minutes ago, Tamesh16 said:

The issue has been identified and a fix is being implemented

@robindb97

it should be up soon

[LukeSavenije]

3 minutes ago, robindb97 said:

sadly discord is still down

 

 

We are working on resolving some internal technical problems now. 

if i have to believe mortis, it's an internal issue and not just limited to Cloudflare, it can literally affect the whole internet

[Guest]

This made for a fun morning.

[Tamesh16]

Just now, Den-Fi said:

This made for a fun morning.

lmao yeah, so many sites went down, i was so confused 

[Jurrunio]

22 minutes ago, VegetableStu said:

i misread that as Outrage, LOL

Aww snap

[colonel_mortis]

1 hour ago, LukeSavenije said:

if i have to believe mortis, it's an internal issue and not just limited to Cloudflare, it can literally affect the whole internet

There were two issues. The first was this global issue, which didn't necessarily affect the whole internet but did affect a number of large providers including Cloudflare and AWS when accessing from some ISPs. The second was an internal Discord issue, which might have been initially caused by the other problem but persisted after that was fixed.

 

 

For those who are curious, the problem here (as I understand it) stems from the fact that the internet is made up of a bunch of different network providers. Each network advertises the IP addresses that it can provide access to, which is usually just the IP addresses within its network and the IPs within the networks that are paying it for the traffic. However, if there is a configuration problem on a network, it might accidentally start advertising that it can reach other networks too. If that route looks better, according to a bunch of heuristics for "better", other networks might start sending traffic there. This traffic might then get immediately dropped because the network doesn't actually know how to get it to its destination, or it might overload some links within the network and result in inconsistent service. The net result is the same though - traffic that takes the other route has problems.

[Tamesh16]

9 minutes ago, colonel_mortis said:

The net result is the same though - traffic that takes the other route has problems.

I didn't really know much of networking so i wasn't able to add in more details of the problem, 

hopefully that doesn't make the post inadequate, Thank you for clarifying though

[rcmaehl]

Was this caused by Cloudflare or was it BGP HIJACKING?

[Tamesh16]

3 minutes ago, rcmaehl said:

BGP HIJACKING?

the article on verge has quote from Cloudflare saying it was "BGP routing leak "

[colonel_mortis]

2 hours ago, rcmaehl said:

Was this caused by Cloudflare or was it BGP HIJACKING?

It was a BGP issue on a third party provider's network, but there's nothing to indicate that it was malicious hijacking - from what I've seen it seems more likely to be a misconfiguration or bug than an attempt at hijacking.