Your Connection Is Not Private - Let's Encrypt are revoking over 3 million TLS certificates

[colonel_mortis]

Sources: Let's Encrypt announcement, Scott Helme's Blog

 

TLS certificates are used by websites to prove that you are talking to the actual website, and to not someone who is intercepting the connection, as part of HTTPS. To make it work there are a number of trusted companies, called certificate authorities, to whom website owners can prove that they are the actual owner of the site; the certificate authority then issues a certificate which basically says "I confirm that the holder of this certificate is the owner of the site". Let's Encrypt is (one of) the largest certificates, and use a completely automated system to issue certificates for free.

 

There are a lot of rules that certificate authorities are required to follow during the certificate issuing process, including that they must check for a CAA record, which is a special DNS record that restricts which certificate authorities are allowed to issue certificates for that domain. The idea of CAA is that if you know that you always get your certificates from DigiCert, you can prevent Let's Encrypt for issuing any certificates for your domain so that you are more protected in the event of a problem at one certificate authority.

 

On the 29th February 2020, Let's Encrypt discovered a bug in their implementation of the CAA check which could a certificate to be issued by Let's Encrypt even if the CAA record prohibits it:

Quote

if a subscriber validated a domain name at time X, and the CAA records for that domain at time X allowed Let’s Encrypt issuance, that subscriber would be able to issue a certificate containing that domain name until X+30 days, even if someone later installed CAA records on that domain name that prohibit issuance by Let’s Encrypt.

 

When the vulnerability was discovered, they immediately stopped issuing certificates and fixed the issue, but any wrongfully issued certificates would be unaffected by that. Therefore, Let's Encrypt are now revoking 3 million certificates that could have experienced this bug (certificates where the CAA check was not performed correctly). From their FAQ:

Quote

Q: How many certificates are affected?
A: 2.6%. That is 3,048,289 currently-valid certificates are affected, out of ~116 million overall active Let’s Encrypt certificates. Of the affected certificates, about 1 million are duplicates of other affected certificates, in the sense of covering the same set of domain names.

 

Everyone whose certificates have been revoked should have been emailed, but if you are concerned then you can check using this online tool, or look up the certificate's serial number in the list of revoked certificates by following Let's Encrypt's instructions.

 

It is very unlikely that anyone has actually abused this vulnerability, but it is absolutely right for Let's Encrypt to revoke those certificates just in case.

 

If a site has had their certificate revoked and they haven't yet reissued it, visitors will get a scary error message when they try to access it. For example, in Chrome it looks like this:

 

 

Certificate authorities rely on being trusted by browsers to operate, and certificate authorities that have had repeated serious violations of the the rules in the past have been distrusted by browsers, effectively forcing them out of business. That is not going to happen in this case, because this was a single incident that affected only one additional layer of security. It is also very reassuring that Let's Encrypt have been so open and proactive about this issue. However, it does raise important concerns about the consequences that other simple bugs in certificate authorities' code could have.

 

Unfortunately there are no practical alternatives to certificate authorities at the moment because entirely decentralising trust is an extremely hard problem, but it will be interesting to see if this leads to more research in that area.

[Crunchy Dragon]

*buzzer sound*
Missing personal input, please update

 

/s

[Guest]

I was so happy that they were so upfront and swift about this.

The last thing we need is to lose faith in Let's Encrypt.

That would be a huge chunk of lost faith in humanity I could never recover.

[FakeCIA]

Well...I'm screwed. I run a couple websites through a third-party publisher that have this coming through. Thought it was just on user's ends until I saw this. I can't do anything on my end through, since I'm not the domain owner or in charge of security. Best I can do is make sure the sites are unpublished until a fix goes live.

[Kisai]

10 minutes ago, colonel_mortis said:

 

Unfortunately there are no practical alternatives to certificate authorities at the moment because entirely decentralising trust is an extremely hard problem, but it will be interesting to see if this leads to more research in that area.

Looks like that push for https everywhere has come back to bite everyone. Put enough eggs in one basket, eventually there will be an accident.

 

One should keep in mind that most CPanel sites use their own CAA and not Let's Encrypt, otherwise that number would be far larger. I got one of the emails for 66% of the servers, and while certbot is on a cronjob, yum is not, and thus one of those servers certbot was refusing to renew until like 20 minutes to expiry, even when told to force. Let's Encrypt clearly dropped the ball here, and thus they need to have a way to "push" a staggered immediate update next time.

 

Because there will be a next time.

[Guest]

7 minutes ago, FakeCIA said:

Well...I'm screwed. I run a couple websites through a third-party publisher that have this coming through. Thought it was just on user's ends until I saw this. I can't do anything on my end through, since I'm not the domain owner or in charge of security. Best I can do is make sure the sites are unpublished until a fix goes live.

Just run the check tool he linked. You'll probably find that yours in unaffected.

[FakeCIA]

1 hour ago, Den-Fi said:

Just run the check tool he linked. You'll probably find that yours in unaffected.

It is. I checked with my third-party and they're getting a crap load of complaints. Luckily, I'm not in charge.

[Trinopoty]

5 minutes ago, Kisai said:

Looks like that push for https everywhere has come back to bite everyone. Put enough eggs in one basket, eventually there will be an accident.

 

One should keep in mind that most CPanel sites use their own CAA and not Let's Encrypt, otherwise that number would be far larger. I got one of the emails for 66% of the servers, and while certbot is on a cronjob, yum is not, and thus one of those servers certbot was refusing to renew until like 20 minutes to expiry, even when told to force. Let's Encrypt clearly dropped the ball here, and thus they need to have a way to "push" a staggered immediate update next time.

 

Because there will be a next time.

Anything and everything comes back to bite in the ass given enough time. Let's Encrypt is no exception.

 

Publishing updates to distribution repositories is a complex and slow process. These updates mostly happen in batches every few days or so.

One way to have an updated copy of certbot is to use the certbot-auto script but that tends to being along it's own issues particularly for RHEL derivatives.

[Guest]

3 minutes ago, FakeCIA said:

It is. I checked with my third-party and they're getting a crap load of complaints. Luckily, I'm not in charge.

Ouch. I managed to come out unscathed. Sucks to see you got hit.

[FakeCIA]

1 hour ago, Den-Fi said:

Ouch. I managed to come out unscathed. Sucks to see you got hit.

Thankfully, I'm not responsible for security and the sites don't collect any information about financial data. I'd be more concerned if it was that, but so far no ones reporting anything wrong or damaging, just complaints.

[LAwLz]

25 minutes ago, Kisai said:

Looks like that push for https everywhere has come back to bite everyone. Put enough eggs in one basket, eventually there will be an accident.

How did it bite anyone in their ass? 

So some users has to get a new cert. Big deal. And in the meantime until they get the new cert (a 5 minute process) they are still more protected than if they were running plain http.

[Kisai]

6 minutes ago, LAwLz said:

How did it bite anyone in their ass? 

So some users has to get a new cert. Big deal. And in the meantime until they get the new cert (a 5 minute process) they are still more protected than if they were running plain http.

Because people were getting the notification to update without enough notice to do anything. So what happens is that anyone who didn't get a notification, or certbot refused to connect, wound up with a dead site.

 

This is the kind of stupidity offered by forcing https-only. If your site doesn't store information from users, then letting them connect via HTTP doesn't matter, and people who complain about mundane sites not being encrypted don't realize it's a zero-sum game for sites with ads on them. They can either run their site entirely unencrypted, or it has to be fully encrypted, because you can't mixed-mode the ads. So older browsers that can't do TLS 1.2, or don't have one of the three remaining ciphers end up with no site to view if there isn't a regular HTTP version to visit. Not even Google is that stupid. The only web browser that doesn't work on google.com is IE6.

 

[SeriousDad69]

1 hour ago, Kisai said:

Because people were getting the notification to update without enough notice to do anything. So what happens is that anyone who didn't get a notification, or certbot refused to connect, wound up with a dead site.

 

This is the kind of stupidity offered by forcing https-only. If your site doesn't store information from users, then letting them connect via HTTP doesn't matter, and people who complain about mundane sites not being encrypted don't realize it's a zero-sum game for sites with ads on them. They can either run their site entirely unencrypted, or it has to be fully encrypted, because you can't mixed-mode the ads. So older browsers that can't do TLS 1.2, or don't have one of the three remaining ciphers end up with no site to view if there isn't a regular HTTP version to visit. Not even Google is that stupid. The only web browser that doesn't work on google.com is IE6.

 

I agree HTTPS doesn't need to be on everything or on every website, but supporting outdated web browsers should be the least of anyone's concern. The main reason I've seen for forcing HTTPS on everything is pervasive Government and Internet Service Provider surveillance.

I guess there just isn't enough of a downside to not HTTPS everything.

[LividPanda]

7 hours ago, Kisai said:

Because people were getting the notification to update without enough notice to do anything. So what happens is that anyone who didn't get a notification, or certbot refused to connect, wound up with a dead site.

 

This is the kind of stupidity offered by forcing https-only. If your site doesn't store information from users, then letting them connect via HTTP doesn't matter, and people who complain about mundane sites not being encrypted don't realize it's a zero-sum game for sites with ads on them. They can either run their site entirely unencrypted, or it has to be fully encrypted, because you can't mixed-mode the ads. So older browsers that can't do TLS 1.2, or don't have one of the three remaining ciphers end up with no site to view if there isn't a regular HTTP version to visit. Not even Google is that stupid. The only web browser that doesn't work on google.com is IE6.

 

A website should only ever use HTTPS, there are no excuses. That site is a liability. That site may be hosted safely but it is going through many corporate, state, and government owned pieces of equipment. Should they be able to inject scripts, ads, modify pages, or add images onto a HTTP page so it looks like the creator of the site did it? No.  Attackers have used HTTP sites to attack other sites, this happens all the time. HTTPS ensures content integrity and also give the person the ability to detect tampering. If you only ever encrypt your secrets they stand out like a sore thumb, encrypt everything. Even if your site doesn't collect any user information HTTPS provides confidentiality for header information, contents, and URLs. Even internal corporate sites or VPN only sites should be using HTTPS. Also just because a sites ads are over HTTP doesn't change the fact that that site should still be using HTTPS. They should figure out a way out of that ad publishing contract and ask why they are serving ads over HTTP.

[LAwLz]

9 hours ago, Kisai said:

Because people were getting the notification to update without enough notice to do anything. So what happens is that anyone who didn't get a notification, or certbot refused to connect, wound up with a dead site.

 

This is the kind of stupidity offered by forcing https-only. If your site doesn't store information from users, then letting them connect via HTTP doesn't matter, and people who complain about mundane sites not being encrypted don't realize it's a zero-sum game for sites with ads on them. They can either run their site entirely unencrypted, or it has to be fully encrypted, because you can't mixed-mode the ads. So older browsers that can't do TLS 1.2, or don't have one of the three remaining ciphers end up with no site to view if there isn't a regular HTTP version to visit. Not even Google is that stupid. The only web browser that doesn't work on google.com is IE6.

 

1) the sites aren't down, even if the cert has been revokes. You can still browse the sites. It's just that visitors gets a warning that the cert is expired. 

 

2) why would encryption only matter if it stored info about the users? HTTPS protects against lots of other things too. For example it prevents anyone from eavesdropping on what you read about. When I'm using https to open this thread, my ISP has no way of knowing which thread I am viewing. It might be able to tell I'm browsing linustechtips.com, but not what I am looking at on the site. That's very important to protect if you want any kind of privacy. 

 

3) you can mix and match http and https content. It's not recommended though. But ads don't have to be http. You can serve ads over https, and many do. This is such a major non-issue its ridiculous to even mention it. 

 

4) Having old ciphers and insecure ciphers allowed on a site is a security risk because of downgrade attacks. "It needs to support older browsers" is a bad argument for why websites should be less secure. Just update your browser. 

[Engineer.AI]

This is not just bad, but really, really horrifying. Why has the hacker meta developed so much, but the internet meta hasn't been able to keep up? There are bots for this particular thing - I mean they use them nearly everywhere else, why not engineer ai for something of this sort. Makes me wanna curl up in anger!

[Pickles von Brine]

Thanks for the heads up!

[Letgomyleghoe]

2 hours ago, Engineer.AI said:

This is not just bad, but really, really horrifying. Why has the hacker meta developed so much, but the internet meta hasn't been able to keep up? There are bots for this particular thing - I mean they use them nearly everywhere else, why not engineer ai for something of this sort. Makes me wanna curl up in anger!

They do engineer AI for this, but the ai is code to so it can be hacked. 

 

And let's be real hackers are always one step ahead, we may have a security update, but the hackers still have another way to break through, and when the site finishes closing up the second hole, the hacker have another

 

Sorry for the bad grammar, it's 2:00 am and I can't fucking sleep.

[Sir Asvald]

Damn. I still use LE for my public facing domains. I use my own CA to encrypt all data internally and some public ones. I know that I can use LE for wildcard SSL. But I don't want keep on updating every 3 months.

[Engineer.AI]

12 minutes ago, scuff gang said:

They do engineer AI for this, but the ai is code to so it can be hacked. 

 

And let's be real hackers are always one step ahead, we may have a security update, but the hackers still have another way to break through, and when the site finishes closing up the second hole, the hacker have another

 

Sorry for the bad grammar, it's 2:00 am and I can't fucking sleep.

Ofcourse I agree 100%. Hackers are always one step ahead, I mean that's part of their job, isn't it?

 

lol, I dont think anybody minds the grammar on this forum

[Kisai]

10 hours ago, LAwLz said:

1) the sites aren't down, even if the cert has been revokes. You can still browse the sites. It's just that visitors gets a warning that the cert is expired. 

 

2) why would encryption only matter if it stored info about the users? HTTPS protects against lots of other things too. For example it prevents anyone from eavesdropping on what you read about. When I'm using https to open this thread, my ISP has no way of knowing which thread I am viewing. It might be able to tell I'm browsing linustechtips.com, but not what I am looking at on the site. That's very important to protect if you want any kind of privacy. 

 

3) you can mix and match http and https content. It's not recommended though. But ads don't have to be http. You can serve ads over https, and many do. This is such a major non-issue its ridiculous to even mention it. 

 

4) Having old ciphers and insecure ciphers allowed on a site is a security risk because of downgrade attacks. "It needs to support older browsers" is a bad argument for why websites should be less secure. Just update your browser. 

1) Try explaining to your clients why their multi-million dollar site is throwing scare warnings at users. The amount of people using let's encrypt is too significant, the same can be said of users using cloudflare. If a business is not willing to spend the money on doing things properly, relying on "free" services is a liability. Users have been trained to equate every browser warning with "this site has been hacked"

 

2) HTTPS protects against nothing except data in transit, and if a user is just viewing an image or a video, that's a complete waste of resources to begin with. Who really gives a care about what comics or porn you are viewing , really. There is no secret AI out there tagging people who have a fetish for boring things. Ultimately only two things require HTTPS: Banks, and anything that you use a credit card/passport with. Everything else is overblown scare-mongering unless you're in a country like PRC, in which case not even HTTPS protects you. If you are in an enterprise network, HTTPS does not protect you from being spied on either. Your ISP has this ability as well.

 

Let me tell you, if your corporate IT insists on you keep using MSIE, it's probably because that's the only browser capable of traversing the transparant HTTPS re-wrapping. When you use Chrome, every website will sit there and look like it's been paused before it loads as the site or file you're trying to load passes through this process, and when you need to download 500MB driver files, it's incredibly annoying.

 

3) No you can't. It's a zero sum game. Mixed mode pages are treated the same as non-https sites by the browser.

 

4) Tell google that. They don't even eat their own dog food.

https://www.ssllabs.com/ssltest/analyze.html?d=www.google.com&s=2607%3af8b0%3a4005%3a802%3a0%3a0%3a0%3a2004&hideResults=on&ignoreMismatch=on

 

We're all back to playing the security theater game with the web browser, and one is incredibly naive to believe that this will not result in a more fragile web ecosystem. 

 

HTTPS has not:

a) stopped any malware from spreading

b) stopped any phishing attacks

c) stopped any viruses from spreading

d) stopped any governments from spying on their citizens

e) stopped any corporations from spying on their staff

f) stopped any ISP's from spying on their customers

g) stopped any websites from spying on their visitors

 

Ultimately what has happened is that where some place has prevented downgrade attacks, they instead just block access completely. It really does come back to either securing your site to the point that you purposely prevent any device older than 5 years from connecting to it, or you leave enough holes in it so that everyone can reach it. So if the option is to either let someone run TLS 1.0 with TLS_RSA_WITH_3DES_EDE_CBC_SHA, or run unencypted, I would rather they run unencypted so they know it's unencyptred than give them the false notion that HTTPS site is protecting anything at all. My content sites may get an A+ on SSLlabs, but forced https only came about because the advertiser network had their own hand forced by Chrome's cookie changes in version 80.

[LAwLz]

18 minutes ago, Kisai said:

1) Try explaining to your clients why their multi-million dollar site is throwing scare warnings at users. The amount of people using let's encrypt is too significant, the same can be said of users using cloudflare. If a business is not willing to spend the money on doing things properly, relying on "free" services is a liability. Users have been trained to equate every browser warning with "this site has been hacked"

But this could have happened with a paid cert as well. In fact, it has happened with a ton of paid certs before.

It being free has no affect on this whatsoever.

Why did you put "free" it quotes anyway?

 

 

19 minutes ago, Kisai said:

2) HTTPS protects against nothing except data in transit, and if a user is just viewing an image or a video, that's a complete waste of resources to begin with.

What do you mean by "data in transit"? You mean when it is being transferred from one computer to another? Well duh, that's when the data is actually vulnerable and needs protection. Saying that HTTPS only protects data in transit is like saying "condoms only protect you while fucking, so don't use them". It makes no sense.

And it's not wasted resources to protect against third parties seeing what resources you're accessing on a website.

 

 

22 minutes ago, Kisai said:

Who really gives a care about what comics or porn you are viewing , really. There is no secret AI out there tagging people who have a fetish for boring things.

Do you honestly don't believe there are AIs or other algorithms out there making profiles on you based on what your browser habits are? Have you been living under a rock the last 5 years? Have you heard of things like PRISM? Oppressive governments trying to censor their citizens?

 

23 minutes ago, Kisai said:

Ultimately only two things require HTTPS: Banks, and anything that you use a credit card/passport with.

That might be your opinion, but I think you'll have a hard time finding anyone serious about security that agrees with you.

The only people who might agree with you are probably things like the Chinese government.

 

25 minutes ago, Kisai said:

If you are in an enterprise network, HTTPS does not protect you from being spied on either. Your ISP has this ability as well.

Ehm, what? Can you please explain how you (wrongly) believe that the network admins on the enterprise network I am on right now can view the HTTPS protected traffic I am sending? The reason why I am confident you won't be able to give me a good answer is because I am the network admin for the enterprise network I am sitting on, and I can not view my own traffic.

 

If you have a little bit of networking knowledge my guess is that your answer will be MITM at the firewall with a root cert installed on all domain joined computers. However, I think that's a very outdated way of doing things which don't really offer that much protection. It doesn't work for BYOD which is becoming more common. It requires MITMing the TSL session which in my opinion opens up a whole can of worms and potential vulnerabilities, and it's just a cat and mouse game at that point. Plus it requires more processing power in the firewalls, something I am trying to cut down on.

 

But a better question is, who do you believe my ISP would be able to view HTTPS traffic from my network? The enterprise network argument  I could kind of get (although I think it's a stupid way to tackle the issue), but not the claim that my ISP can somehow break the encryption. Sounds like something you just made up.

 

 

36 minutes ago, Kisai said:

Let me tell you, if your corporate IT insists on you keep using MSIE, it's probably because that's the only browser capable of traversing the transparant HTTPS re-wrapping. When you use Chrome, every website will sit there and look like it's been paused before it loads as the site or file you're trying to load passes through this process, and when you need to download 500MB driver files, it's incredibly annoying.

What are you on about? What exactly do you mean when you say "capable of traversing the transparant HTTPS re-wrapping"? I don't understand what you are saying.

Also, I think you've encountered some issue and incorrectly puts the blame on HTTPS.

And no, my work does not insist on using Internet Explorer, and if your work does so then it's probably not for the reason you stated. The reason why some corporations insist you use IE are because:

1) It integrates fairly well with GPOs and AD, which means they get more control over your computer and settings.

2) Some sites like for example older versions of SharePoint only works with IE. Or they contain ActiveX elements in them (which only works in IE).

3) It means less troubleshooting for the IT staff if something goes wrong. Fewer programs to support => Fewer things to keep track off => Easier troubleshooting.

 

 

41 minutes ago, Kisai said:

3) No you can't. It's a zero sum game. Mixed mode pages are treated the same as non-https sites by the browser.

I think you're misusing the term "zero sum game" there, or maybe I am not understanding what you're referring to.

What do you mean when you say mixed mode pages are "treated the same as non-https sites by the browser"? Do you think a mixed mode page is completely unencrypted? Because that's just flat out wrong.

 

 

Replies to the rest later...

[Kisai]

7 hours ago, LAwLz said:

 

But a better question is, who do you believe my ISP would be able to view HTTPS traffic from my network? The enterprise network argument  I could kind of get (although I think it's a stupid way to tackle the issue), but not the claim that my ISP can somehow break the encryption. Sounds like something you just made up.

 

It's not breaking the encryption, it's transparently unwrapping and re-wrapping the TLS session. If your enterprise, or ISP doesn't want you to visit something (eg like you're in China) you don't get around it. It's like squishing an ant with a sledgehammer, but that's exactly what enterprises use.

 

https://www.okta.com/sites/default/files/Okta Technical Security Whitepaper.pdf

 

On more than one occasion, to be prompted to login to the SSO just to download a driver, or to let some stupid application download an update. Then it's not directly downloaded to you, something is downloading it somewhere to be scanned before it's sent to you.

[colonel_mortis]

3 minutes ago, Kisai said:

It's not breaking the encryption, it's transparently unwrapping and re-wrapping the TLS session. If your enterprise, or ISP doesn't want you to visit something (eg like you're in China) you don't get around it. It's like squishing an ant with a sledgehammer, but that's exactly what enterprises use.

 

https://www.okta.com/sites/default/files/Okta Technical Security Whitepaper.pdf

 

On more than one occasion, to be prompted to login to the SSO just to download a driver, or to let some stupid application download an update. Then it's not directly downloaded to you, something is downloading it somewhere to be scanned before it's sent to you.

ISPs rewrapping TLS is precisely what the system of certificates and certificate authorities protects against. At enterprises the company will install their own root certificate authority on all of their managed devices so that the certificate that they present when they intercept the traffic will be accepted, but if you used the network with an unmanaged device (without their root CA installed) you would just get TLS errors.

 

The scheme mentioned in your post is not for TLS at all, it is a delegated authentication provider. However, a scheme similar to what I think you were aiming for is used by Cloudflare, with the key difference being that the website owners explicitly give Cloudflare access to the domain so that Cloudflare can issue a certificate on their behalf. Cloudflare is not able to issue a certificate for a site that isn't using their services.

 

The reason that there are rules for certificate authorities, like the ones that Let's Encrypt inadvertently broke in this case, are to ensure that the only person who can obtain a certificate that will be trusted by unmanaged browsers is the owner of the website. An ISP, malicious attacker, or government agency could only be issued a valid certificate for someone else's website if the CAB rules are violated, and it is no longer possible to do so without detection because in order to be valid, a certificate has to also be submitted to a publicly available log (certificate transparency).

 

I may be wrong, but it is my understanding that the Great Firewall of China doesn't do any TLS decryption, it just blocks certain services outside of China based on DNS or IP blacklists. If it did do decryption, that would require all devices in China to be installed with a root Chinese government certificate authority.

[leadeater]

1 hour ago, colonel_mortis said:

ISPs rewrapping TLS is precisely what the system of certificates and certificate authorities protects against. At enterprises the company will install their own root certificate authority on all of their managed devices so that the certificate that they present when they intercept the traffic will be accepted, but if you used the network with an unmanaged device (without their root CA installed) you would just get TLS errors.

Channel binding can also be used to protect against SSL inspection, the connecting client sends the public key hash it's been given to the destination and that system compares the hash to it's own public hash and if they are different the connection is rejected, because there is a man in the middle. The same check is done in the other direction, both ends check the hashes. It's not widely used but it's something I have to deal with for RDS Gateways and LDAP servers etc, I have to worry about it because of load balancing services and SSL offloading (need to use the same key pair on the LB and the back-end service). Token binding is being proposed for introduction in to the wider TLS standards, would make SSL inspection even on enterprise networks much harder. This and HTTPS DNS are being quite a challenge.