Categories
Transport Layer Security (TLS)

Are CBC cipher suites still used in HTTPS?

This is a question I’ve asked myself about two months ago. The question was whether I should follow Mozilla’s Guidelines for Server Side TLS and throw old, legacy ciphers (like CBC-based ciphers), overboard. This depends on if they’re still used in practice. There will certainly always be clients that only support legacy stuff. I have no interest in maintaining absurd backwards compatiblity by sacrifcing security. But I do want some reasonable amount of backwards compatibilty – I certainly don’t want to lock out legitimate users using only slightly outdated software.

I’m already requiring HTTPS on my sites since 2016. In 2018 I dropped support for TLS 1.1 and below – TLS 1.2 was published in 2008, so if you’re TWELVE years behind: Sorry, but that isn’t going to end well.

In the past I’ve also killed of more legacy ciphers such as those using Cipher Block Chaining Mode (CBC). I’ve now reenabled these for testing purposes. The CBC mode has some serious issues: For one, there are (possible) padding oracle attacks against CBC. This isn’t just a theoretical thing, actual attacks were performed in the past. While we’re at it: Don’t forget the Poodle variants… There are even more dragons lurking around, such as Lucky Thirteen (a timing side channel based attack on CBC). But it doesn’t stop here: CBC (in combination with TLS) has also suffered from indirect attacks like OpenSSL’s 0-Length Bug. There’s also a whole bunch of issues with ciphers that separate encryption and authentication. As a conclusion, ciphersuites that use CBC aren’t really preferable in a modern, secure encryption scheme.

TLS 1.2 introduced AEAD (authenticated encryption) ciphers. Those do both things (authentication & encryption) together, eliminating many oracle issues. They don’t solve all issues, but as of now (2020) they’re the best we have at this time. Because TLS 1.2 is already 12 years old, these secure AEAD ciphersuites have seen great adoption in practice.

Now, Mozilla recommends to turn of CBC entirely and only use those AEAD ciphersuites. I would love to do that, but before I turn my back on CBC I want to be sure that all common clients do support the newer AEAD things. Qualys SSL Labs seems to suggest that many outdated macOS computers do not support any AEAD cipher (Note that we do not care about clients that do not even speak TLS 1.2 – that’s already off the table).

Because I couldn’t find any usage study on cipher suites used in the wild, I decided to do my own, non-scientific study. For the past two months I logged all TLS ciphersuites used by clients that performed at least one complete HTTPS request [to my server]. This means that most TLS scans, which do not make any HTTP(S) requests, were excluded from the logging. This way most data gathered is from actually meaningful clients (crawlers & humans plus a few additional bots).

I’m actively monitoring whether I see clients that really do not speak anything better than CBC. For the past two months, I haven’t had one (except for two clients which were definetly TLS scanners). However, at this point I’m still not done with the survey. I will continue to monitor for some more and once I have gathered sufficient data, I will decide whether to ditch CBC or not.

Note that this is not a scientific study of any kind, and as such I will not compile any beautfiul data sets. I may post some stats here in the future if I feel like it. I’m not getting paid for this, so involved effort is low.

If you want to see what my current TLS setups looks like, you can see for yourself on Qualys. I’m also planning on writing a more detailed blog post about how the setup looks and the reasoning behind it. It’s all on my TODO list, I promise.

Update (1 month later)

I have continued to monitor ciphersuites and I did have some clients sending HTTP requests using CBC ciphers. All of these clients were some type of TLS scanner or vulnerability scanners. I couldn’t identify a single useful bot or human user with CBC. As a result I have turned of CBC completly. That also means that server cipher preference is now off and the client can choose it’s preferred cipher (as Mozilla recommends).

Another Update (many months later)

I did encounter some very old clients now only supporting CBC (but TLS 1.2) while setting up some new services. Those were IOT devices (sigh) and other embedded legacy stuff like webbrowsers from Smart TV’s and similar. One of these didn’t even knew what elliptic curves are (no ECDHE and no ECDSA). I’ve reenabled a single CBC cipher suite for those clients.

Another major problem with these clients is the upcoming switch of Let’s Encrypt’s Root Certificate, ISRG Root X1. Those clients are so old that they don’t have an up to date trust store and thus cannot validate certificates from the new root. I initially underestimated how many legacy client are out there, even my own household has some devices affected by this. Sadly, there’s no good strategy here, but that’s a topic for another post…

Categories
Transport Layer Security (TLS)

Hidden openssl.cnf options and PrioritizeChaCha

Blarg, another long title. Again. Sorry.

So, this is something that I actually discovered a while (months) ago, so my memories are already a bit less fresh, but I think I still remember the important things.

I discovered this, because I had a common TLS CipherSuite config problem: I needed a server-site cipher order (e.g server has it’s own preferences), because I had some legacy weaker cipher enabled and we obviously don’t want clients with incorrect cipher order to connect with a potentially weak suite, when there are better suites available. But I also wanted to use ChaCha (more on that later).

If you don’t understand what I’m talking about, below is a summary on how TLS cipher suite negotiation works. If you know that stuff already, skip this chapter.

TLS cipher suite negotiation


TLS negotiates a cipher suite on each connection. There are many available – some are really secure, others are “okay-ish”, others are really bad. The really bad ones are usually disabled either at server or client-side (or both). But the “okay-ish” suites are generally enabled both at server and client side, even though client and server potentially also support the strong “good” ones.
What happens is that the client gives the server a list of cipher suites, which should be sorted in order of preference, and the server chooses one of them, depending on what is supported by both.

The server has generally two ways of choosing:

  • Server preference. The server maintains its own list of “preferred” cipher suites and chooses the best one on its list, depending on what the client supports.
  • Client preference. The server just chooses the first suite of the clients list that both parties support.

The thing with Client preference is, that there are (or were) some clients that send incorrectly ordered cipher suite lists – they have insecure ciphers at the top and more secure ones at the bottom. If the server lets the client choose, a weak cipher suite will be negotiated, even though both parties may support something stronger.

On the other hand, if the server maintains a correctly sorted list, one can guarantee that with server preference, the server will choose the most secure option depending on what the client supports. Thats the reason why server preference is the most common setting in real-world TLS.

Just use server cipher order and be done with it!

Yeah, that’s what many people do. And it was fine for some time. The thing is, TLS 1.2 at some point introduced a new cipher / cipher suite:

TLS_CHACHA20_POLY1305_SHA2561

That is an entirely new encryption algorithm (ChaCha with 20 rounds) and Poly1305 authentication. While those may not be exactly new, they were newly introduced into TLS.

What’s so special about ChaCha?

I won’t go into details here, I’m sure there are already posts elsewhere that cover this. Basically, ChaCha is (probably) secure and fast. This is interesting for machines that are not AES hardware accelerated (no AES-NI). This applies for example to most ARM based systems, like smartphones. Other examples may be IOT or embedded devices. Those don’t have accelerated AES and ChaCha is noticeably faster on those devices, especially when comparing against AES-256. Again, I won’t post big benchmarks here, those are elsewhere.

In summary, we would really want to use ChaCha on such devices.

Then just set ChaCha as preferred server cipher and be done with it!

But… I still like AES! Because that’s the thing: While ChaCha may be the new cool thing, many devices do have AES-NI (hardware accelerated AES) and thus can do AES faster than ChaCha. Plus, AES has been around for quite a while and so far we consider it secure. It’s also FIPS and similar certified, for people that are into that stuff.

So we still want to use AES! We just don’t want to use it with devices that have no AES-NI.

How can you get the best of both worlds?

What we need is something like this: If the client says “I don’t have AES-NI”, use ChaCha. If the client doesn’t speak ChaCha, or if it has AES-NI, use AES.

But how to determine if a client doesn’t have AES-NI? Fortunately, the standards made this not that hard: Modern clients that speak ChaCha will put ChaCha on the top of their cipher list, if they don’t have AES-NI.

Older clients, or clients with AES-NI will have something else on top. So the easiest way to use ChaCha with clients is to let the clients choose: If they have ChaCha on the top, use that. If not, use whatever else they have at the top. But that puts us back to the original problem above! What if an old, broken client puts legacy stuff on the top? There was a reason why we used server cipher preference.

Okay, so we want both client AND server cipher preference. That’s sadly something that OpenSSL can’t do. But OpenSSL developers saw the issue and offered us a solution:

OpenSSL has a flag for this!

Yep, that’s right. OpenSSL has a flag, called “PrioritizeChaCha” that does exactly what I described above: It will choose ChaCha if the client says “thats my most preferred cipher suite”, but will still honor the server cipher order in all other cases.

However, that is generally a compile-time flag. Meaning that you need to compile OpenSSL with this option if you want it. This isn’t really what I consider ideal. Many people use pre-compiled packages – after all, most distros ship programs this way. Especially when talking about security relevant stuff like OpenSSL, I personally like relying on the debian security team to update important packages. I just don’t favor the idea of compiling OpenSSL myself – it’s a lot more work for me, with not much benefits.

There’s another way: Setting this without re-compiling OpenSSL!

Yup, and that’s what this blog post was supposed to be about. The entire text wall above was just introduction for this.

OpenSSL has a config file. If you didn’t know that, I don’t blame you: I didn’t either, until last year or so. Since OpenSSL is mostly used as a library, there isn’t much to configure. On Debian-like systems, the file is /etc/ssl/openssl.cnf. The majority of this file is related to Certificate Authority settings. That’s because you can make your own CA with OpenSSL and that file is for persistently setting parameters needed for a (more or less) real CA.

But you can set more than just CA-specific parameters! The file can do more. Documentation has been sparse previously, but has been improved a lot recently. You can find documentation for it here. Thanks to Geert for pointing this out.

I will now try to give some guidance for that file. Ignore all the CA stuff – I’m not going to explain that. I’m interested in what this file can do to modify the TLS behavior.

Basically, the file is structured in sections, which are formatted like this: [section_name] followed by a list of settings valid for that section. A section ends when the next section starts.

In order to configure TLS options, we need to look at what sections are reponsible for that. The starting point is the setting openssl_conf which is part of some kind of “root section” that has no specific name. The default value of openssl_conf is set to default_conf. In order to define what default_conf should be, you can define a section called default_conf that sets everything you want.

Since we’re interested in TLS (SSL) specific settings, we set the ssl_conf parameter in our custom default_conf to the name of a section, let’s say ssl_sect (short for ssl_section). In the ssl_sect we then declare a setting system_default which holds some TLS options, unless the application overrides them explicitly. We set system_default to system_default_sect, again that is short for “system default section”.

In this section, we finally get to set our TLS/SSL parameters. You can for example define custom TLS min/max versions, default cipher suites and most importantly flags like PrioritizeChaCha.

[default_conf]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
MinProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=2
Options = PrioritizeChaCha,NoRenegotiation

The example above sets default parameters for each application that doesn’t specify anything else on it’s own:

  • Minimum TLS version is set to TLS 1.2
  • Some insecure cipher suites are disabled (security level is set to two). See the SECLEVEL documentation here and note the exact defintions of them tend to change between (major) OpenSSL releases.
  • Two flags are set: PrioritizeChaCha and NoRenegotation

The important things here are the flags: PrioritizeChaCha is explained in the beginning of this post. This config would enable the flag by default for all applications using OpenSSL. The other flag, NoRenegotation, disables TLS renegotiation. That’s mostly “for fun”. Renegotation was broken in the past, has not much use and was even killed in TLS 1.3, so not much reason to have it on in TLS 1.2 either.

Since I feel like this post is already way longer than intended, I will stop here. If you have any further questions about some of the topics covered here, feel free to use the comment section below. Or email me directly – my email is here somewhere.

Categories
Transport Layer Security (TLS)

Monitoring certificate issuance with the power of certificate transparency

Wow, what a long title. And so big! I really need to rework this site layout – or think of shorter titles.

Basics first, what is certificate transparency?

It’s a (still new) technology to monitor the issuance of certificates. (Wow what a great statement, that’s exactly the title of this post). No really, the thing is that the TLS protocol (and many other security protocols) have always had trust issues. Not psychological, but technical.

“On the Internet, nobody knows you’re a dog”

– Peter Steiner

And that’s where trust issues start. The protocols that make up the internet at it’s core do not make any guarantees about who you’re talking with. That’s a problem if you want security – if I want to talk to my banking website, how can I be certain that it’s actually my bank at the other side of the wire? Could be anyone.

In order to solve that problem, digital certificates were introduced. But to be honest, those didn’t solve the trust issue, but just shifted it. Certificates are basically a proof that you’re who you claim you are. In order to make them worth something, they need to get some kind of notarization. Those are done by certificate authorities. They are magical instances which never make mistakes and only issue certificates to good persons with a proven identity. That’s why everyone trusts these certificate authorities and that’s how we solved trust issues on the internet.

Sounds too good to be true? In fact it actually works surprisingly well. But not always. Certificate authorities (CA) make mistakes, get hacked or otherwise compromised and then you got the problem:
Because everyone trusts the CA and the power of a CA is hardly limited (there’s CAA, but we’ll keep that to another post), a fraudulent CA can pretty much issue certificates for every domain it wants.

That is obviously a bad thing and there should be protection against that – and there is! One way of protection is certificate transparency.

You still haven’t told what that is…

We’ll get there. The idea behind Certificate Transparency is a requirement that all certificates must be logged publicy. If you can force every CA to always and without exception log every certificate that it issues, you can at least detect immediatly when a CA goes full rogue – with smart (and automatic) monitoring you can also detect other abnormalities in the certificate ecosystem, but that’s way beyond the scope of this post.

And how does it work exactly?

Sadly, the details are also way beyond the scope of this post. To make it short, Google Chrome (and other browsers, most notably Apple products) requires every freshly issued certificate to appear in a public log that records all certificates. If a certificate isn’t logged there, the browser does not allow a connection to the website. That’s how you force every CA to log every certificate issuance there. There’s a lot of crypto involved to make sure that certificates can prove they’re part of a log and the logs use a Merkle tree to prove they’re also doing fair-play.

I’m not a certificate authority. If it works, why should I care about certificate transparency?

The thing is, certificate transparency is pretty worthless if it isn’t fully utilized. If every certificate is put into a log, but no one watches that log, the log is useless. There are a few people and companies that monitor all logs for unusual behavior and notify relevant authorities if something happens. But that isn’t enough to detect every attack. Small (e.g. single domain) attacks cannot be detected by large-scale monitoring. That only works if domain owners do their own monitoring.

Note that certificate transparency logs are public. That means anyone is free to query them if they want to. That’s something that we can utilize. If you’re a domain owner, you can setup an automatic monitor application that tells you whenever someone issued a certificate for your domain. Imagine someone took over your domain unnoticed – maybe due to a leak, DNS hijack or something else. If an attacker has (limited or full) control over your domain, they can most likely also issue a fake certificate for the domain and thus even intercept encrypted communications via a man in the middle attack.

Certificate Transparency can help in detecting such attacks.

If you have a monitor that tells you who has issued certificates for your domain(s) recently, you get notified immediatly that something’s wrong. That in itself doesn’t fix your security hole, but at least aids in early detection and possibly mitigation of further breaches.

That sounds fine, but where/how can I monitor my domain(s)?

The ideal way would be to query all certificate transparency logs yourself. But that’s a complicated and resource-intensive thing, which we will not cover here.

Luckily, there’s a “cheap alternative”. Theres a service, hosted by Sectigo (formerly Comodo), called crt.sh. They automatically query all approved certificate transparency logs and put every certificate in a public database, that can be nicely accessed – both via web and via PostgreSQL.

If you want automatic monitoring for your domains, you can just access that website (or database) periodically, fetch every certificate for your domain and look for new ones. And that is it! That’s what basic monitoring is. You can get pretty sophisticated monitoring, sure, but a simple email notification should already do the trick for most people. You will probably need a human though to determine if a new certificate issuance was “legal” (remember, Let’s Encrypt certificates are renewed/re-issued every 60 days) or fraudulent (you can in theory automate this, but it’s not that easy).

crt.sh protip: When querying the web service of crt.sh, putting the parameter output=json in a query will get you results in nice JSON format – much easier for scripts. For example, if you want all certificates for %.germancoding.com (% means “match anything”) you could do this:
https://crt.sh/?q=%25.germancoding.com&output=json

If you go to advanced search you will see some more web filter options (like excluding expired certificates). For implementation details, or if you want to access the database directly, it’s probably best to head over to their forums and look for answers there – most things are explained there.

Tutorial time: Simple monitor python script

The following is an example script that does the following:

  • Fetch all currently valid certficates from crt.sh for a few given example domains (germancoding.com and a few more in this example)
  • Bail out if that fails, but have automatic retries too
  • Load a local SQLite database from disk (“knowncerts.db”) that stores all certificates that we already know
  • Compare list of certificates from crt.sh with the local database, look for new certificates
  • If one or more new certificates are found:
    • Try to fetch additional information about them
    • Then send an email to a preconfigured address, telling them about the new certificate(s)
    • Finally, add the new certificate(s) to the local database

Known caveats and other things to know about the script:

  • The script relies on some python-dependencies (subprocess, requests, sqlite, urllib3, and json) being available (as well as python itself of course)
  • The script relies on some OS commands being available:
    • ‘openssl’ command in order to parse a new certificate
    • A working ‘mail’ command in order to send emails
  • The local SQLite database ‘knowncerts.db’ must already exist locally. In order to create the database, you need to do the following:
    • Make sure sqlite/sqlite3 is installed (including the python module)
    • sqlite3 knowncerts.db will create a new database (or open an existing one)
    • Executing CREATE TABLE IF NOT EXISTS certs (id text, serial text); in the freshly opened SQLite-terminal will create the neccessary table structure expected by the script.
    • Type .exit to get out of that terminal.

Now, let’s finally get to it. The formatting below is a lot different than the rest of this page, but this snippet really needs as much space as it can get.

import subprocess
from urllib3.util.retry import Retry
import requests
from requests.adapters import HTTPAdapter
import sqlite3
from subprocess import Popen, PIPE
import json

# Replace email with your own
email = "your-email@example.com"

# Replace domain(s) below with your own
# The OR syntax is understood by crt.sh and searches
# multiple domains
domains = "%.germancoding.com OR example.com OR letsencrypt.org"

# Setup a retry system in case crt.sh has temporary hiccups
session = requests.Session()
retry = Retry(
   total=10,
   status=10,
   status_forcelist=[429],
   backoff_factor=1,
   respect_retry_after_header=True,
)
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

payload = {'Identity': domains, 'exclude': 'expired', 'output': 'json'}
r = session.get('https://crt.sh/', params=payload)
r.raise_for_status()
allcerts = r.json()

changes = False
conn = sqlite3.connect('knowncerts.db')
c = conn.cursor()
c.execute("SELECT id FROM certs");
knowncerts = c.fetchall()
knowncerts = [i[0] for i in knowncerts]
knownserials = []
for item in allcerts:
   id = str(item['id'])
   if id not in knowncerts:
      domain = item['name_value'].strip().splitlines()[0]
      text = "ERROR UNDEFINED TEXT"
      serial = "ERROR"
      sendmail = True
      try:
        # Get PEM cert from crt.sh and print it using OpenSSL
        payload = {'d': id}
        r = session.get('https://crt.sh/', params=payload)
        r.raise_for_status()
        pem = r.text
        domain = subprocess.run(["openssl", "x509", "-subject", "-noout"], input=pem, text=True, check=True, capture_output=True, timeout=5).stdout.replace("subject=", "").strip().splitlines()[0].replace("CN = ", "")
        serial = subprocess.run(["openssl", "x509", "-serial", "-noout"], input=pem, text=True, check=True, capture_output=True, timeout=5).stdout
        serial = serial.replace("serial=", "").strip().splitlines()[0]
        c.execute('SELECT id, serial FROM certs WHERE serial=?', [serial])
        if len(c.fetchall()) > 0 or serial in knownserials:
           #print("debug: serial already known")
           sendmail = False
        text = subprocess.run(["openssl", "x509", "-text", "-noout"], input=pem, text=True, check=True, capture_output=True, timeout=5).stdout
      except Exception as e:
        text = "(Failed to fetch additional data - " + repr(e) + ")"
        pass
      c.execute("INSERT INTO certs VALUES(?, ?)", [id, serial]);
      if sendmail:
         mailbody = "Issued cert can be found at https://crt.sh/?id=" + id + "\n\n\n" + text
         subprocess.run(["mail", "-s", "New cert " + domain + " for a monitored domain", email], input=mailbody, text=True, check=True, timeout=60)
      knowncerts.append(id)
      knownserials.append(serial)
      changes = True
if changes:
  conn.commit()
c.close()
conn.close()

You can put the script above in a certnotify.py python file and run it with a cronjob, hourly or so:

# crontab -e

27 * * * * python3 /path/to/certnotify.py >> /var/log/certnotify/certnotify.log 2>> /var/log/certnotify/certnotify.err.log