Cloudflare Reverse Proxies are Dumping Uninitialized Memory

[vorticalbox]

full list of almost 4.3 million domains can be found here

https://github.com/pirate/sites-using-cloudflare/blob/master/README.md

 

or a website to check can be found

http://www.doesitusecloudflare.com/

 

Quote

It became clear after a while we were looking at chunks of uninitialized memory interspersed with valid data. The program that this uninitialized data was coming from just happened to have the data I wanted in memory at the time. That solved the mystery, but some of the nearby memory had strings and objects that really seemed like they could be from a reverse proxy operated by cloudflare - a major cdn service.

 

A while later, we figured out how to reproduce the problem. It looked like that if an html page hosted behind cloudflare had a specific combination of unbalanced tags, the proxy would intersperse pages of uninitialized memory into the output (kinda like heartbleed, but cloudflare specific and worse for reasons I'll explain later). My working theory was that this was related to their "ScrapeShield" feature which parses and obfuscates html - but because reverse proxies are shared between customers, it would affect *all* Cloudflare customers.

We fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major cloudflare-hosted sites from other users. Once we understood what we were seeing and the implications, we immediately stopped and contacted cloudflare security.

 

 

 

This is about as bad as it gets,they've potentially been spraying it into caches all across the Internet. Tavis found it by accident just looking through Google search results, that's how bad this situation was.

 

potential every website that uses cloudflare has potentially compromised everything that is being served; API keys, sessions, personal information, user passwords, the works.

 

i don't even know where to start in thinking of cleaning up this mess but I will be changing my passwords for all cloudflare sites.

 

Sources

https://bugs.chromium.org/p/project-zero/issues/detail?id=1139

 

https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

 

[zMeul]

at least we know about it

[sazrocks]

Well, crap. Almost everything uses cloud fare. Is there any info on how long this has been happening?

[2FA]

20 minutes ago, vorticalbox said:

i don't even know where to start in thinking of cleaning up this mess but I will be changing my passwords for all cloudflare sites.

I'd be more worried about the fact that they were even transmitting plaintext passwords instead of the hash.

[N3v3r3nding_N3wb]

This website uses Cloudland, does it not?

[samiscool51]

2 hours ago, vorticalbox said:

i don't even know where to start in thinking of cleaning up this mess but I will be changing my passwords for all cloudflare sites.

the LTT forums is one of the many sites that use cloudfare...

maybe now people will stop using it and use something else.

[Zodiark1593]

The password I use for this site isn't one of my good ones anyway. 

[arnavvr]

2 minutes ago, Zodiark1593 said:

The password I use for this site isn't one of my good ones anyway. 

I use 1Password for this exact reason.

[Zodiark1593]

30 minutes ago, arnavvr said:

I use 1Password for this exact reason.

*ponders whether to test it for lackluster amusement*

 

Looking at the article, it is quite impressive to see such a serious bug fixed in so little time, while not breaking much else doing so. 

[pspfreak]

https://arstechnica.com/security/2017/02/serious-cloudflare-bug-exposed-a-potpourri-of-secret-customer-data/

 

Quote

Cloudflare, a service that helps optimize the security and performance of more than 5.5 million websites, warned customers today that a recently fixed software bug exposed a range of sensitive information that could have included passwords, and cookies and tokens used to authenticate users.

The leakage was the result of a bug in an HTML parser chain Cloudflare uses to modify Web pages as they pass through the service's edge servers. The parser performs a variety of tasks, such as inserting Google Analytics tags, converting HTTP links to the more secure HTTPS variety, obfuscating email addresses, and excluding parts of a page from malicious Web bots. When the parser was used in combination with three Cloudflare features—e-mail obfuscation, server-side Cusexcludes, and Automatic HTTPS Rewrites—it caused Cloudflare edge servers to leak pseudo random memory contents into certain HTTP responses.

Basically you should pretty much change your password for pretty much everything, in which I would assume Linus Tech Tips as well since LTT uses Cloudflare.

[RGProductions]

FFS now i have to think of another generic password

[The Sloth]

2 minutes ago, RGProductions said:

FFS now i have to think of another generic password

Hunter2 or *******

[deleted_member_030719]

... for fucksakes... I literally just switched to using CloudFlare in January and now there's a data leak. 

[arnavvr]

1 hour ago, Zodiark1593 said:

*ponders whether to test it for lackluster amusement*

 

Looking at the article, it is quite impressive to see such a serious bug fixed in so little time, while not breaking much else doing so. 

You should try it, there is a one month free trial.

[Guest]

8 minutes ago, RGProductions said:

FFS now i have to think of another generic password

Good let me remember that you use Another Generic as your password

[Theo J]

3 minutes ago, Jed M said:

... for fucksakes... I literally just switched to using CloudFlare in January and now there's a data leak. 

Jed, holy shit haha. I know you from MCM. (It's supertolerator.)

 

Anyway, that's pretty crazy. I'll most likely be changing my passwords. Thanks for the heads up OP.

[deleted_member_030719]

2 minutes ago, Theo J said:

Jed M, holy shit haha. I know you from MCM. (It's supertolerator.)

 

Anyway, that's pretty crazy. I'll most likely be changing my passwords. Thanks for the heads up OP.

Off topic, but I got banned from there because someone refunded a payment and I wouldn't resend it to them lmao.

[Theo J]

1 minute ago, Jed M said:

Off topic but, I got banned from there because someone refunded a payment and I wouldn't resend it to them lmao.

That sucks. Good to see you again though.

[Smile]

1 hour ago, Zodiark1593 said:

The password I use for this site isn't one of my good ones anyway. 

LTT forum supports 2 factor authentication.

[imreloadin]

From the Article:

 

Quote

Graham-Cummings, the Cloudflare CTO, has ruled out the possibility that secret keys for customers' transport layer security certificates were exposed in the leaks. Still, he said end-user passwords, authentication cookies, OAuth tokens used to log into multiple website accounts, and encryption keys Cloudflare uses to protect server-to-server traffic were all at risk of being exposed.

quite the leak indeed...

 

Edit: So I just sifted through that wonderful 71MB text file and found that 3 websites I use are on there. Those being Udemy.com Nexusmods.com and Newegg.com

 

Figured I'd add them in case anyone else uses them here, we all probably have Newegg accounts

Edited February 24, 2017 by imreloadin
Added websites.

[kirashi]

5 hours ago, vorticalbox said:

i don't even know where to start in thinking of cleaning up this mess but I will be changing my passwords for all cloudflare sites.

I'd love to do this, but I'd be here all week. At what point do consumers get reimbursement for time wasted due to poorly engineered services?

 

Full stop: I am a developer myself, so I completely understand this is no singular person's fault. However, when literally all services are passed through some form of Web Application Firewall in the future, at what point do consumers have a choice in the protection and security of their data? We can't opt out of using WAF's, since the whole point is to protect the sites that hide behind them, but at the same time, what do consumers do to protect themselves from a single point of failure? More importantly, who pays for the security of our data?

 

According to ArsTechnica @sazrocks it could have been happening as far back as September 16th 2016.

[2FA]

36 minutes ago, kirashi said:

According to ArsTechnica  it could have been happening as far back as September 16th 2016.

Cloudfare says the September 22nd in their blog post when Automatic HTTP Rewrites was enabled, January 30th 2017 for Server Side Excludes, February 23th for Email Obfuscation. All of which have been implicated in the cause. They know the particular line of code that caused it but they aren't sure exactly which feature was the root cause.

[kirashi]

1 hour ago, DeadEyePsycho said:

Cloudfare says the September 22nd in their blog post when Automatic HTTP Rewrites was enabled, January 30th 2017 for Server Side Excludes, February 23th for Email Obfuscation. All of which have been implicated in the cause. They know the particular line of code that caused it but they aren't sure exactly which feature was the root cause.

Good that they know the exact dates. I keep seeing different information everywhere.

[2FA]

7 minutes ago, kirashi said:

Good that they know the exact dates. I keep seeing different information everywhere.

Here's the direct source. https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

 

It was literally as simple as using the '==' operator in an if statement condition instead of the correct '>='.

[LAwLz]

Everyone should change their passwords regardless of whether or not the sites you use have 2 factor authentication. Having your password out there is bad.

 

And I will take this opportunity to shill recommend Keepass2.

No need to pay money for a password manager when there is a great one that's free and open source out there.