Categories
Server Administration Transport Layer Security (TLS)

Let’s Encrypt and expired root certificates

Let’s Encrypt is soon going to do something that I personally call “root-cross-signing”. This has definetly impact on some TLS clients, which is why I decided to give it some attention.

But first, let’s explain what the potential problem is: For a certificate to be trusted, it needs to be signed by a trusted Certificate Authority (CA) – the shift-of-trust model I already mentioned in earlier posts. Traditionally, the trust anchor – the point where the trust starts – used to be something called a root certificate1. Those are not signed by anyone (well, they technically sign themselves, but that doesn’t really do that much) and are shipped with your TLS client, or operating system (or similar). We call those certificates that are shipped with your client the trust store.

A root certificate usually signs at least one intermediate certificate (which in turn can sign other – lower – intermediates) and the lowest intermediate certificate signs the leaf certificate. That is the certificate of the actual service you’re connecting to, e.g this blog. This collection of certificates is what we call a certificate chain2. The server sends this chain to the client and the client then verifies this chain, checking whether all signatures are correct and whether the client trusts at least one (or the highest) of those certificates (remember, the clients has a trust store that contains root certificates)3. At least in theory. In practice this is much more difficult, as for example modern browsers to things like caching trust anchors + intermediates or auto-fetching intermediates, there is more than one chain, there is more than one root, chains are send with incorrect ordering and much more.

The actual issue

So, what has all of this to do with Let’s Encrypt?

As they tell us in their blog, they’re soon going to use a chain that looks like this: leaf -> R3 -> ISRG Root X1 -> DST Root CA X3. They’re going to use this chain for years to come. But DST Root CA X3 expires later this year! What happens then?

What will happen is heavily client specific, and depends on how your client verifies certificate chains. Some clients will not have a problem, others will completly break.

Let’s Encrypt tried to downplay the issue a bit, stating that Android compability is important and only “very old” other clients will have issues. I believe this isn’t the entire truth and I want to have a closer look at compatibility across clients, hence this blog post.

Why are we even doing this? Why not just go with ISRG Root X1?

As Let’s Encrypt explains, this is due to the fact that so many devices, especially Android, have older trust stores that do not get updates. Those devices have DST Root CA X3 in their trust store, but not ISRG Root X1. Hence the magic dance to try to support those old devices. But sadly, this will break compatibility with newer devices. Lets have a closer look at compatibility.

Compatibility across clients

Android

The new chain was specifically created for Android devices (>= 4.0) and thus no compatibility issues are expected on those Androids.

OpenSSL

OpenSSL has really bad chain validation in older versions and will thus try to validate up to the highest certificate in the chain, even if it should know that a lower certificate is a root and could just stop validating there. This means that older versions of OpenSSL will break with Let’s Encrypts new default chain, even if the trust store is up to date.

Affected is OpenSSL version less than 1.1 4. Yes, OpenSSL 1.1.0 was technically released 4 years ago, but many, many computers I know still ship something older. Take for example Ubuntu 16.04 which is supported until 2024 – that distro still uses OpenSSL 1.0.2g and thus needs the manual fix (see footnote above/below) in order to connect with Let’s Encrypt subscriber servers starting late 2021. Many people who need FIPS certification still use 1.0, because the FIPS module isn’t available in OpenSSL 1.1. I don’t have statistics about the used versions in the wild, but I expect millions of OpenSSL versions that aren’t yet running 1.1+ out here. Most of these will have trouble if no one does anything.

GnuTLS

A library I often see when I work with Debian/Ubuntu related things, or anything else that doesn’t like OpenSSL for some reason. Those products often tend to use GnuTLS, as its FOSS.

GnuTLS will break with the new Let’s Encrypt chain starting late 2021, unless you’re running a version newer or equal to 3.6.14 (released 2020). Again, lets have a look at what the distributions ship:

  • Debian 10 Buster currently ships GnuTLS 3.6.7, which is < 3.6.14, so it’s probably affected. Note that this is the latest Debian release, as of now there is nothing newer (but Debian Bullseye will be released this summer, which will have a fixed version). However, many people will run buster – or even older versions, you still see many stretch installations in the wild – for years to come. The GnuTLS versions here will probably all be affected. I’m not aware of a workaround for GnuTLS.
    Update: Numerous distributions have started to ship backports/workarounds to affected GnuTLS versions.
  • For Ubuntu, apparently everything older than Ubuntu 20.10 (Groovy Gorilla) is affected. This means that even the current LTS release Ubuntu 20.04 LTS “Focal Fossa”, supported until 2025+, is most likely affected (they apparently ship GnuTLS 3.6.13, one version below the fix).
    Update: Ubuntu recently backported patches to supported versions.

In summary, many distributions that don’t have a very fast version cycle will ship slightly older GnuTLS versions, and everything that is older than a year will likely break.

LibreSSL

Another library I sometimes see in the wild, especially when OpenBSD is involved. I’m not that familiar with it, so I’m going to make it short.

Everything less than 3.2.0 is affected, which was released in June 2020. So you need some version that is less than a year old to be not affected.

Windows, macOS and browsers native TLS libraries

Let’s Encrypt says those are not affected – at least someone who is fine.

Conclusion

I believe that Let’s Encrypts decision to use an Android-compatible chain by default will break setups for many people. That doesn’t mean that it was the wrong call, but it means that people need to be aware that their scripts, API integrations or whatever non-browser-non-android they have will most likely break.

What can I do if I don’t want all of this?

You can manually configure the alternate chain Let’s Encrypt will offer later this year. This means that you’re breaking Android compatiblity with Android < 7.1, but will preserve compatibility with all others, if their trust store contains ISRG Root X1 (which is a hidden requirement for all non-Android devices starting in late 2021, this applies for both chains).

Further reading

I highly recommend Ryan Sleevies Implementation Showdown, it explains in a much greater detail about how implementations did it wrong and it also gives you an idea what clients may be affected (it covers many more clients than those mentioned in this post).

Sources

Details about affected version numbers were taken from here:

Categories
Transport Layer Security (TLS)

Why 100% isn’t always the best score

Please do not try to reach a 100% score on SSLabs.

I’ve seen this numerous times on the Let’s Encrypt forums and so I felt the need to scream out: Please don’t do this.

A small introduction to SSLabs and it’s ratings

The website SSLabs provides a service that checks SSL/TLS hosts for their security level, scans for typical TLS issues and displays a final score upon finishing scanning.

Users not familiar with the inner workings of TLS probably don’t understand much of the displayed data. It is therefore natural for these users to interpret the data they understand most – the score. The score is displayed at the top of the results page and may look like this:

Screenshot taken from my own blog, on 18.09.2020 @ dev.ssllabs.com
Screenshot taken from my own blog, on 18.09.2020 @ dev.ssllabs.com

As you can see here, there’s an “overall rating”. The highest possible value is A+, the lowest rating is F (or various connection error states). Besides the rating, there’s also a percentage score (0 – 100%), divided into four categories:

  • Certificate – The score of the site’s certificate
  • Protocol support – A score, based on which protocol versions are supported by the server
  • Key Exchange – Score based on the strength of all available key exchanges
  • Cipher strength – Score based on the strength of all available cipher suites

If you look at my screenshot, you should notice that I’ve got 100% only in two categories – Certificate and Protocol Support. The other two categories are only at 90%. Why is that, and why shouldn’t I fix that?

Before we dive into the details, the reader should be aware that the rating system of SSLabs constantly changes. Future versions may work differently and this information may not be up to date.

Category 1 – the certificate

Getting 100% is pretty simple here, and most users don’t need to do anything to achieve this. According the the SSLabs rating guide, the certificate is evaluated by typical misconfigurations or weaknesses, and points are deducted if anything is out of the usual. Most CA’s (like Let’s Encrypt) will generally not issue “bad” certificates, so there’s not much room for issues here. There are some things to misconfigure though, e.g a bad or incomplete cert chain – SSLabs will show a warning in the results if something was found.

Category 2 – protocol support

We could talk ages about what SSL/TLS version exist, and which you should use or shouldn’t use, but I’m going to make it short: Trust me, or trust Mozilla’s Guidelines, or trust one of the million sites that have great scores on SSLabs:

Use TLSv1.2 and TLSv1.3 only. Do not enable anything lower (anything newer doesn’t exist at the time of writing). Doing this will also get you the 100% score on SSLabs in terms of protocol support. TLSv1.2 is twelve years old, so compatibility isn’t a factor unless you need to support extremly old things (in that case you should consider isolating the legacy systems).

Category 3 – key exchange

Now it gets interesting. This is one of two categories where I don’t have 100%. And you shouldn’t either. In order to get 100% on key exchange, you would need to make sure that all algorithms used in the key exchange have a theoretical bit security strength that is greater than some arbitrary value set by SSLabs. That is currently at least 4096 Diffie-Hellman, or a 384-bit EC curve or higher. You’re not allowed to offer anything with less size, that would decrease your score (usually to 90%, which is what you see on my site).

So, why are larger keys bad?

They’re not bad, but their increase in security is mostly theoretical. We cannot break a 256 bit EC key, and we also cannot break a 384 bit key. We may be able to do this in the future though, which is why some folks tend to prefer larger keys. However, some researchers say that it is much more likely that a curve is broken by a technical flaw, rather than by true brute-force power. If such a flaw is found, it will most likely affect both the smaller and the larger curves. The value in practice is small.

The downside of a larger key is however, that not all clients support it (also performance). Google Chrome, for example, has dropped support for one the largest NIST curves, secp521r1. You cannot use this curve together with Chromium-based browser, which is the majority of the web. Android 7.0 has a bug, where the only supported elliptic curve is prime256v1. Prime256v1 is one of the smaller curves (256 bits), but statistically the most used ECDSA curve in the public internet.

So, using larger curves will not really increase your strength against an attacker, but will lock out clients without reason. That is why I’m reiterating: A DHE key exchange of 2048-bit is still okay today, and you should also offer smaller curves. You obviously can offer larger values – but do not drop support for the smaller ones yet. Except for things like 1024-bit RSA/DHE1 – that’s dangerously low. But 2048-bit to 4096-bit RSA/DHE is a reasonable range to offer.

Category 4 – cipher strength

The second category where I only have 90%. The reasoning is very similar to category 3 – a higher score kills of clients support, without a strong increase in security level.

Getting 100% here requires all offered block ciphers to have a key length of 256 bits or greater. Anything lower must not be offered. The most famous (and used) block cipher is probably AES. The largest key size available for AES is 256, the lowest 128 bits.

Similar to key exchanges, more bits are not neccessarily more secure. A larger key means more work brute-force wise and generally increases computation work for many attacks. But, there are also attacks, like Cache-timing attacks, that can work even on the large AES-256 keys. So yes, a larger key is a bit better, but it is also a lot slower and security doesn’t increase dramatically.

The major issue again here is, besides performance, that not all clients talk AES256 (in all cipher combinations). For example, according to SSLLabs, Firefox 47 on Windows 7 only speaks AES128-GCM, not AES256-GCM. 256-bit AES-CBC is supported, but we don’t want to talk CBC. So requiring the use of AES256 will again lock out lots of clients.

It is therefore far better to offer both – AES128-GCM and AES256-GCM. If the client supports both, you can still select AES256 by turning on server-side cipher preference (see my older posts on TLS). But do not force it, unless you like killing of clients.

I hope this post has cleared up some misconceptions about the scores on SSLabs, and why a full 100%-score is sacrificing support without better overall security. I also want to remind server operators, that a secure TLS connection is only a part of the deal – a secure site also requires a well-engineered and maintained application behind it. I plan to make some more web-related posts about typical attacks, security headers and similar things in the future.

If you have any questions, feel free to ask below or contact me directly – my email is here somewhere.

Categories
Transport Layer Security (TLS)

Monitoring certificate issuance with the power of certificate transparency

Wow, what a long title. And so big! I really need to rework this site layout – or think of shorter titles.

Basics first, what is certificate transparency?

It’s a (still new) technology to monitor the issuance of certificates. (Wow what a great statement, that’s exactly the title of this post). No really, the thing is that the TLS protocol (and many other security protocols) have always had trust issues. Not psychological, but technical.

“On the Internet, nobody knows you’re a dog”

– Peter Steiner

And that’s where trust issues start. The protocols that make up the internet at it’s core do not make any guarantees about who you’re talking with. That’s a problem if you want security – if I want to talk to my banking website, how can I be certain that it’s actually my bank at the other side of the wire? Could be anyone.

In order to solve that problem, digital certificates were introduced. But to be honest, those didn’t solve the trust issue, but just shifted it. Certificates are basically a proof that you’re who you claim you are. In order to make them worth something, they need to get some kind of notarization. Those are done by certificate authorities. They are magical instances which never make mistakes and only issue certificates to good persons with a proven identity. That’s why everyone trusts these certificate authorities and that’s how we solved trust issues on the internet.

Sounds too good to be true? In fact it actually works surprisingly well. But not always. Certificate authorities (CA) make mistakes, get hacked or otherwise compromised and then you got the problem:
Because everyone trusts the CA and the power of a CA is hardly limited (there’s CAA, but we’ll keep that to another post), a fraudulent CA can pretty much issue certificates for every domain it wants.

That is obviously a bad thing and there should be protection against that – and there is! One way of protection is certificate transparency.

You still haven’t told what that is…

We’ll get there. The idea behind Certificate Transparency is a requirement that all certificates must be logged publicy. If you can force every CA to always and without exception log every certificate that it issues, you can at least detect immediatly when a CA goes full rogue – with smart (and automatic) monitoring you can also detect other abnormalities in the certificate ecosystem, but that’s way beyond the scope of this post.

And how does it work exactly?

Sadly, the details are also way beyond the scope of this post. To make it short, Google Chrome (and other browsers, most notably Apple products) requires every freshly issued certificate to appear in a public log that records all certificates. If a certificate isn’t logged there, the browser does not allow a connection to the website. That’s how you force every CA to log every certificate issuance there. There’s a lot of crypto involved to make sure that certificates can prove they’re part of a log and the logs use a Merkle tree to prove they’re also doing fair-play.

I’m not a certificate authority. If it works, why should I care about certificate transparency?

The thing is, certificate transparency is pretty worthless if it isn’t fully utilized. If every certificate is put into a log, but no one watches that log, the log is useless. There are a few people and companies that monitor all logs for unusual behavior and notify relevant authorities if something happens. But that isn’t enough to detect every attack. Small (e.g. single domain) attacks cannot be detected by large-scale monitoring. That only works if domain owners do their own monitoring.

Note that certificate transparency logs are public. That means anyone is free to query them if they want to. That’s something that we can utilize. If you’re a domain owner, you can setup an automatic monitor application that tells you whenever someone issued a certificate for your domain. Imagine someone took over your domain unnoticed – maybe due to a leak, DNS hijack or something else. If an attacker has (limited or full) control over your domain, they can most likely also issue a fake certificate for the domain and thus even intercept encrypted communications via a man in the middle attack.

Certificate Transparency can help in detecting such attacks.

If you have a monitor that tells you who has issued certificates for your domain(s) recently, you get notified immediatly that something’s wrong. That in itself doesn’t fix your security hole, but at least aids in early detection and possibly mitigation of further breaches.

That sounds fine, but where/how can I monitor my domain(s)?

The ideal way would be to query all certificate transparency logs yourself. But that’s a complicated and resource-intensive thing, which we will not cover here.

Luckily, there’s a “cheap alternative”. Theres a service, hosted by Sectigo (formerly Comodo), called crt.sh. They automatically query all approved certificate transparency logs and put every certificate in a public database, that can be nicely accessed – both via web and via PostgreSQL.

If you want automatic monitoring for your domains, you can just access that website (or database) periodically, fetch every certificate for your domain and look for new ones. And that is it! That’s what basic monitoring is. You can get pretty sophisticated monitoring, sure, but a simple email notification should already do the trick for most people. You will probably need a human though to determine if a new certificate issuance was “legal” (remember, Let’s Encrypt certificates are renewed/re-issued every 60 days) or fraudulent (you can in theory automate this, but it’s not that easy).

crt.sh protip: When querying the web service of crt.sh, putting the parameter output=json in a query will get you results in nice JSON format – much easier for scripts. For example, if you want all certificates for %.germancoding.com (% means “match anything”) you could do this:
https://crt.sh/?q=%25.germancoding.com&output=json

If you go to advanced search you will see some more web filter options (like excluding expired certificates). For implementation details, or if you want to access the database directly, it’s probably best to head over to their forums and look for answers there – most things are explained there.

Tutorial time: Simple monitor python script

The following is an example script that does the following:

  • Fetch all currently valid certficates from crt.sh for a few given example domains (germancoding.com and a few more in this example)
  • Bail out if that fails, but have automatic retries too
  • Load a local SQLite database from disk (“knowncerts.db”) that stores all certificates that we already know
  • Compare list of certificates from crt.sh with the local database, look for new certificates
  • If one or more new certificates are found:
    • Try to fetch additional information about them
    • Then send an email to a preconfigured address, telling them about the new certificate(s)
    • Finally, add the new certificate(s) to the local database

Known caveats and other things to know about the script:

  • The script relies on some python-dependencies (subprocess, requests, sqlite, urllib3, and json) being available (as well as python itself of course)
  • The script relies on some OS commands being available:
    • ‘openssl’ command in order to parse a new certificate
    • A working ‘mail’ command in order to send emails
  • The local SQLite database ‘knowncerts.db’ must already exist locally. In order to create the database, you need to do the following:
    • Make sure sqlite/sqlite3 is installed (including the python module)
    • sqlite3 knowncerts.db will create a new database (or open an existing one)
    • Executing CREATE TABLE IF NOT EXISTS certs (id text, serial text); in the freshly opened SQLite-terminal will create the neccessary table structure expected by the script.
    • Type .exit to get out of that terminal.

Now, let’s finally get to it. The formatting below is a lot different than the rest of this page, but this snippet really needs as much space as it can get.

import subprocess
from urllib3.util.retry import Retry
import requests
from requests.adapters import HTTPAdapter
import sqlite3
from subprocess import Popen, PIPE
import json

# Replace email with your own
email = "your-email@example.com"

# Replace domain(s) below with your own
# The OR syntax is understood by crt.sh and searches
# multiple domains
domains = "%.germancoding.com OR example.com OR letsencrypt.org"

# Setup a retry system in case crt.sh has temporary hiccups
session = requests.Session()
retry = Retry(
   total=10,
   status=10,
   status_forcelist=[429],
   backoff_factor=1,
   respect_retry_after_header=True,
)
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

payload = {'Identity': domains, 'exclude': 'expired', 'output': 'json'}
r = session.get('https://crt.sh/', params=payload)
r.raise_for_status()
allcerts = r.json()

changes = False
conn = sqlite3.connect('knowncerts.db')
c = conn.cursor()
c.execute("SELECT id FROM certs");
knowncerts = c.fetchall()
knowncerts = [i[0] for i in knowncerts]
knownserials = []
for item in allcerts:
   id = str(item['id'])
   if id not in knowncerts:
      domain = item['name_value'].strip().splitlines()[0]
      text = "ERROR UNDEFINED TEXT"
      serial = "ERROR"
      sendmail = True
      try:
        # Get PEM cert from crt.sh and print it using OpenSSL
        payload = {'d': id}
        r = session.get('https://crt.sh/', params=payload)
        r.raise_for_status()
        pem = r.text
        domain = subprocess.run(["openssl", "x509", "-subject", "-noout"], input=pem, text=True, check=True, capture_output=True, timeout=5).stdout.replace("subject=", "").strip().splitlines()[0].replace("CN = ", "")
        serial = subprocess.run(["openssl", "x509", "-serial", "-noout"], input=pem, text=True, check=True, capture_output=True, timeout=5).stdout
        serial = serial.replace("serial=", "").strip().splitlines()[0]
        c.execute('SELECT id, serial FROM certs WHERE serial=?', [serial])
        if len(c.fetchall()) > 0 or serial in knownserials:
           #print("debug: serial already known")
           sendmail = False
        text = subprocess.run(["openssl", "x509", "-text", "-noout"], input=pem, text=True, check=True, capture_output=True, timeout=5).stdout
      except Exception as e:
        text = "(Failed to fetch additional data - " + repr(e) + ")"
        pass
      c.execute("INSERT INTO certs VALUES(?, ?)", [id, serial]);
      if sendmail:
         mailbody = "Issued cert can be found at https://crt.sh/?id=" + id + "\n\n\n" + text
         subprocess.run(["mail", "-s", "New cert " + domain + " for a monitored domain", email], input=mailbody, text=True, check=True, timeout=60)
      knowncerts.append(id)
      knownserials.append(serial)
      changes = True
if changes:
  conn.commit()
c.close()
conn.close()

You can put the script above in a certnotify.py python file and run it with a cronjob, hourly or so:

# crontab -e

27 * * * * python3 /path/to/certnotify.py >> /var/log/certnotify/certnotify.log 2>> /var/log/certnotify/certnotify.err.log