Jump to content

[Resolved] Another Cloudflare outage, this time internally caused?

brwainer
By brwainer
in Tech News
 Share
Posted

Many websites are showing “502 Bad Gateway” due to an issue with Cloudflare. This affects floatplane.com, and even cloudflare.com and support.cloudflare.com. Users on twitter report problems with Discord, Feedly, and many websites.

Source for specific sites: various Twitter posts

Source for outage: https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr

Quote

Cloudflare is observing network performance issues. Customers may be experiencing 502 errors while accessing sites on Cloudflare.

We are working to mitigate impact to Internet users in this region.

The list of affected regions on that page looks to be nearly global.

 

My take: Unlike the last outage which was caused by a third party ISP creating a bad BGP route and hijacking traffic, this one appears to be due to a misconfiguration or mistake on Cloudflare’s internal network, based on the error code given for all affected sites.

 

EDIT: 14:15UTC (10:15 AM EST) Cloudflare has posted:

Quote

Cloudflare has implemented a fix for this issue and is currently monitoring the results. 

We will update the status once the issue is resolved.

Hopefully in a few hours we’ll get more details on what went wrong.

Looking to buy GTX690, other multi-GPU cards, or single-slot graphics cards: 

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Discord dependency on Cloudflare sucks, sob.

Personal Desktop":

CPU: Intel Core i7 10700K @5ghz |~| Cooling: bq! Dark Rock Pro 4 |~| MOBO: Gigabyte Z490UD ATX|~| RAM: 16gb DDR4 3333mhzCL16 G.Skill Trident Z |~| GPU: RX 6900XT Sapphire Nitro+ |~| PSU: Corsair TX650M 80Plus Gold |~| Boot:  SSD WD Green M.2 2280 240GB |~| Storage: 1x3TB HDD 7200rpm Seagate Barracuda + SanDisk Ultra 3D 1TB |~| Case: Fractal Design Meshify C Mini |~| Display: Toshiba UL7A 4K/60hz |~| OS: Windows 10 Pro.

Luna, the temporary Desktop:

CPU: AMD R9 7950XT  |~| Cooling: bq! Dark Rock 4 Pro |~| MOBO: Gigabyte Aorus Master |~| RAM: 32G Kingston HyperX |~| GPU: AMD Radeon RX 7900XTX (Reference) |~| PSU: Corsair HX1000 80+ Platinum |~| Windows Boot Drive: 2x 512GB (1TB total) Plextor SATA SSD (RAID0 volume) |~| Linux Boot Drive: 500GB Kingston A2000 |~| Storage: 4TB WD Black HDD |~| Case: Cooler Master Silencio S600 |~| Display 1 (leftmost): Eizo (unknown model) 1920x1080 IPS @ 60Hz|~| Display 2 (center): BenQ ZOWIE XL2540 1920x1080 TN @ 240Hz |~| Display 3 (rightmost): Wacom Cintiq Pro 24 3840x2160 IPS @ 60Hz 10-bit |~| OS: Windows 10 Pro (games / art) + Linux (distro: NixOS; programming and daily driver)
Link to comment
Share on other sites
Link to post
Share on other sites
Posted
2 minutes ago, Princess Luna said:

Discord dependency on Cloudflare sucks, sob.

Forum semi dependency on Cloudflare sucks. :C

PLEASE QUOTE ME IF YOU ARE REPLYING TO ME

Desktop Build: Ryzen 7 2700X @ 4.0GHz, AsRock Fatal1ty X370 Professional Gaming, 48GB Corsair DDR4 @ 3000MHz, RX5700 XT 8GB Sapphire Nitro+, Benq XL2730 1440p 144Hz FS

Retro Build: Intel Pentium III @ 500 MHz, Dell Optiplex G1 Full AT Tower, 768MB SDRAM @ 133MHz, Integrated Graphics, Generic 1024x768 60Hz Monitor


 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
3 minutes ago, Princess Luna said:

Discord dependency on Cloudflare sucks, sob.

now I have to work at work ;-;

 

Thats why Teamspeak is better :^)

if you want to annoy me, then join my teamspeak server ts.benja.cc

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
1 minute ago, rcmaehl said:

Forum semi dependency on Cloudflare sucks. :C

I wonder how much of the internet is dependant on Cloudflare :C

New to Star Citizen? Look no further!

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Is 'IsItDownRightNow.com' also dependent on Cloudfare, because that is not working/slow for me too :/ Kind of awkward when you want to see if I am the issue or the website I try to visit.

"We're all in this together, might as well be friends" Tom, Toonami.

 

mini eLiXiVy: my open source 65% mechanical PCB, a build log, PCB anatomy and discussing open source licenses: https://linustechtips.com/topic/1366493-elixivy-a-65-mechanical-keyboard-build-log-pcb-anatomy-and-how-i-open-sourced-this-project/

 

mini_cardboard: a 4% keyboard build log and how keyboards work: https://linustechtips.com/topic/1328547-mini_cardboard-a-4-keyboard-build-log-and-how-keyboards-work/

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
Just now, The Benjamins said:

now I have to work at work ;-;

Yeah me too, now only have LTT left to kill time along side work.

 

If only more people used Telegram :/

Personal Desktop":

CPU: Intel Core i7 10700K @5ghz |~| Cooling: bq! Dark Rock Pro 4 |~| MOBO: Gigabyte Z490UD ATX|~| RAM: 16gb DDR4 3333mhzCL16 G.Skill Trident Z |~| GPU: RX 6900XT Sapphire Nitro+ |~| PSU: Corsair TX650M 80Plus Gold |~| Boot:  SSD WD Green M.2 2280 240GB |~| Storage: 1x3TB HDD 7200rpm Seagate Barracuda + SanDisk Ultra 3D 1TB |~| Case: Fractal Design Meshify C Mini |~| Display: Toshiba UL7A 4K/60hz |~| OS: Windows 10 Pro.

Luna, the temporary Desktop:

CPU: AMD R9 7950XT  |~| Cooling: bq! Dark Rock 4 Pro |~| MOBO: Gigabyte Aorus Master |~| RAM: 32G Kingston HyperX |~| GPU: AMD Radeon RX 7900XTX (Reference) |~| PSU: Corsair HX1000 80+ Platinum |~| Windows Boot Drive: 2x 512GB (1TB total) Plextor SATA SSD (RAID0 volume) |~| Linux Boot Drive: 500GB Kingston A2000 |~| Storage: 4TB WD Black HDD |~| Case: Cooler Master Silencio S600 |~| Display 1 (leftmost): Eizo (unknown model) 1920x1080 IPS @ 60Hz|~| Display 2 (center): BenQ ZOWIE XL2540 1920x1080 TN @ 240Hz |~| Display 3 (rightmost): Wacom Cintiq Pro 24 3840x2160 IPS @ 60Hz 10-bit |~| OS: Windows 10 Pro (games / art) + Linux (distro: NixOS; programming and daily driver)
Link to comment
Share on other sites
Link to post
Share on other sites
Posted
Just now, Princess Luna said:

Yeah me too, now only have LTT left to kill time along side work.

 

If only more people used Telegram :/

Nah, just hop onto adhoc IRC servers in times of Crisis.

PLEASE QUOTE ME IF YOU ARE REPLYING TO ME

Desktop Build: Ryzen 7 2700X @ 4.0GHz, AsRock Fatal1ty X370 Professional Gaming, 48GB Corsair DDR4 @ 3000MHz, RX5700 XT 8GB Sapphire Nitro+, Benq XL2730 1440p 144Hz FS

Retro Build: Intel Pentium III @ 500 MHz, Dell Optiplex G1 Full AT Tower, 768MB SDRAM @ 133MHz, Integrated Graphics, Generic 1024x768 60Hz Monitor


 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
3 minutes ago, Minibois said:

Is 'IsItDownRightNow.com' also dependent on Cloudfare, because that is not working/slow for me too :/ Kind of awkward when you want to see if I am the issue or the website I try to visit.

https://builtwith.com/isitdownrightnow.com

 

BuiltWith says yes :) 

New to Star Citizen? Look no further!

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Yeah, this website wasn't accessible 5 minutes ago lol. 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

On the incident page they've added a summary of what they know about the issue so far

Quote

We saw a massive spike in CPU that caused primary and secondary systems to fall over. We shut down the process that was causing the CPU spike. Service restored to normal within ~30 minutes. We’re now investigating the root cause of what happened.

For one process to kill half the internet is pretty bad.

HTTP/2 203

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
11 minutes ago, colonel_mortis said:

For one process to kill half the internet is pretty bad.

Agreed, and it's a bit unclear if the only reason we're back online is due to their automatic fall-over. Or if the severe performance impairment WAS the secondary/BCP systems, and we're back online with their primary systems now. 

 

Either way, there's a lot of room for improvement. In my industry (treasury management), a 30 minute production issue isn't half bad, but our outages can regularly dip into several hours long. ?

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Imagine owning bitcoin and seeing it was $26
 

image.png.a1750c882c9130e1b7635461f9abf7ec.png

PLEASE QUOTE ME IF YOU ARE REPLYING TO ME

Desktop Build: Ryzen 7 2700X @ 4.0GHz, AsRock Fatal1ty X370 Professional Gaming, 48GB Corsair DDR4 @ 3000MHz, RX5700 XT 8GB Sapphire Nitro+, Benq XL2730 1440p 144Hz FS

Retro Build: Intel Pentium III @ 500 MHz, Dell Optiplex G1 Full AT Tower, 768MB SDRAM @ 133MHz, Integrated Graphics, Generic 1024x768 60Hz Monitor


 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
1 hour ago, Princess Luna said:

If only more people used Telegram :/

Ayyyy another telegram user!

 

Also this thread is hilarious (in a 12 year old childish way) with the "cloud to butt" extension for google chrome

 

3 minutes ago, rcmaehl said:

Imagine owning bitcoin and seeing it was $26

 

Shame they didn't let you buy it at 26$

Donate your unused computer resources for cancer and other research. For Zinsey ❤

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
1 minute ago, iamdarkyoshi said:

Its owned by facebook. Which is disgusting. 

 

They also just straight up copy telegram's open source client code IIRC

Wasn't telegram created by the Devs of WhatsApp?

One day I will be able to play Monster Hunter Frontier in French/Italian/English on my PC, it's just a matter of time... 4 5 6 7 8 9 years later: It's finally coming!!!

Phones: iPhone 4S/SE | LG V10 | Lumia 920 | Samsung S24 Ultra

Laptops: Macbook Pro 15" (mid-2012) | Compaq Presario V6000

Other: Steam Deck

<>EVs are bad, they kill the planet and remove freedoms too some/<>

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
1 minute ago, suicidalfranco said:

Wasn't telegram created by the Devs of WhatsApp?

Quote

Telegram was started in 2013 by the brothers Nikolai and Pavel Durov, the creators of VK, the largest social network of Russia. Telegram Messenger LLP is an independent nonprofit company located in Berlin, Germany, that is not connected to VK.

I think we're derailing the thread though

Donate your unused computer resources for cancer and other research. For Zinsey ❤

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
Just now, iamdarkyoshi said:

I think we're derailing the thread though

The more you know

One day I will be able to play Monster Hunter Frontier in French/Italian/English on my PC, it's just a matter of time... 4 5 6 7 8 9 years later: It's finally coming!!!

Phones: iPhone 4S/SE | LG V10 | Lumia 920 | Samsung S24 Ultra

Laptops: Macbook Pro 15" (mid-2012) | Compaq Presario V6000

Other: Steam Deck

<>EVs are bad, they kill the planet and remove freedoms too some/<>

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
53 minutes ago, iamdarkyoshi said:

.

Without mentioning WhatsApp annoying need for the smartphone to always be online making the stand alone Desktop application less useful :/

 

Any ways we're back! Good thing it didn't extend today, gotta access all those Cloudflare depended domains :3

Personal Desktop":

CPU: Intel Core i7 10700K @5ghz |~| Cooling: bq! Dark Rock Pro 4 |~| MOBO: Gigabyte Z490UD ATX|~| RAM: 16gb DDR4 3333mhzCL16 G.Skill Trident Z |~| GPU: RX 6900XT Sapphire Nitro+ |~| PSU: Corsair TX650M 80Plus Gold |~| Boot:  SSD WD Green M.2 2280 240GB |~| Storage: 1x3TB HDD 7200rpm Seagate Barracuda + SanDisk Ultra 3D 1TB |~| Case: Fractal Design Meshify C Mini |~| Display: Toshiba UL7A 4K/60hz |~| OS: Windows 10 Pro.

Luna, the temporary Desktop:

CPU: AMD R9 7950XT  |~| Cooling: bq! Dark Rock 4 Pro |~| MOBO: Gigabyte Aorus Master |~| RAM: 32G Kingston HyperX |~| GPU: AMD Radeon RX 7900XTX (Reference) |~| PSU: Corsair HX1000 80+ Platinum |~| Windows Boot Drive: 2x 512GB (1TB total) Plextor SATA SSD (RAID0 volume) |~| Linux Boot Drive: 500GB Kingston A2000 |~| Storage: 4TB WD Black HDD |~| Case: Cooler Master Silencio S600 |~| Display 1 (leftmost): Eizo (unknown model) 1920x1080 IPS @ 60Hz|~| Display 2 (center): BenQ ZOWIE XL2540 1920x1080 TN @ 240Hz |~| Display 3 (rightmost): Wacom Cintiq Pro 24 3840x2160 IPS @ 60Hz 10-bit |~| OS: Windows 10 Pro (games / art) + Linux (distro: NixOS; programming and daily driver)
Link to comment
Share on other sites
Link to post
Share on other sites
Posted
4 hours ago, Princess Luna said:

Discord dependency on Cloudflare sucks, sob.

 

4 hours ago, rcmaehl said:

Forum semi dependency on Cloudflare sucks. :C

 

Cloudflare sucks.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Update:

Quote

The cause of this outage was deployment of a single misconfigured rule within the Cloudflare Web Application Firewall (WAF) during a routine deployment of new Cloudflare WAF Managed rules. The intent of these new rules was to improve the blocking of inline JavaScript that is used in attacks. Unfortunately, one of these rules contained a regular expression that caused CPU to spike to 100% on our machines worldwide. This 100% CPU spike caused the 502 errors that our customers saw. At its worst traffic dropped by 82%. We were seeing an unprecedented CPU exhaustion event, which was novel for us as we had not experienced global CPU exhaustion before. At 1402 UTC we understood what was happening and decided to issue a ‘global kill’ on the WAF Managed Rulesets, which instantly dropped CPU back to normal and restored traffic. That occurred at 1409 UTC

 

Source

PLEASE QUOTE ME IF YOU ARE REPLYING TO ME

Desktop Build: Ryzen 7 2700X @ 4.0GHz, AsRock Fatal1ty X370 Professional Gaming, 48GB Corsair DDR4 @ 3000MHz, RX5700 XT 8GB Sapphire Nitro+, Benq XL2730 1440p 144Hz FS

Retro Build: Intel Pentium III @ 500 MHz, Dell Optiplex G1 Full AT Tower, 768MB SDRAM @ 133MHz, Integrated Graphics, Generic 1024x768 60Hz Monitor


 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
1 minute ago, rcmaehl said:

Update:

 

Source

A single? so nice to know about half the internet has no redundency.

I live in misery USA. my timezone is central daylight time which is either UTC -5 or -4 because the government hates everyone.

into trains? here's the model railroad thread!

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
Just now, will4623 said:

A single? so nice to know about half the internet has no redundency.

No one:
Cloudflare: Let's test changes in production!

PLEASE QUOTE ME IF YOU ARE REPLYING TO ME

Desktop Build: Ryzen 7 2700X @ 4.0GHz, AsRock Fatal1ty X370 Professional Gaming, 48GB Corsair DDR4 @ 3000MHz, RX5700 XT 8GB Sapphire Nitro+, Benq XL2730 1440p 144Hz FS

Retro Build: Intel Pentium III @ 500 MHz, Dell Optiplex G1 Full AT Tower, 768MB SDRAM @ 133MHz, Integrated Graphics, Generic 1024x768 60Hz Monitor


 

Link to comment
Share on other sites
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing Tech News

×