Tag: OpenSSL

Categories
Linux Server Administration

UEFI Secure Boot, Debian, DKMS and the R8125B

Recently, I have been playing with UEFI secure boot and custom modules. I am one of the lucky people to own one of the discontinued Odroid H2+ models, a small board with decent computing capabilities.

Odroid H2+ mainboard with components fitted.
https://wiki.odroid.com/odroid-h2/start

The Odroid features an x86 CPU, so there are no ARM weirdnesses to worry about. However, the default BIOS setting of this board seems to be UEFI Secure Boot off. But what even is Secure Boot?

Secure Boot is a method to prevent Rootkits and other malicious software to execute code during boot. While the system is booting, it is generally less protected. If the operating system has not yet loaded, any protection provided by the OS is not yet working. The earlier something runs in the boot process, the more power it has: It can control all execution after itself.

While Secure Boot also has weaknesses and it sometimes debated, it is supposed to alleviate some of these concerns. UEFI Secure Boot ensures that all code part of the boot sequence is authenticated and has not been tampered with.

Visual flow of UEFI verification. BIOS, shim, Grub, kernel and module signatures are validated by the UEFI Secure Boot system.

This graphic means that everything involved during the boot processes must be signed by a trusted certificate. We’re assuming Debian here. By default, most mainboards (UEFI’s) only ship Microsoft’s Root UEFI certificate. Most first-party bootloaders will therefore be signed with Microsoft’s key, to be compatible everywhere.

However, non-Microsoft projects use their own infrastructures and don’t want/can’t have every update signed by Microsoft. Instead, they use their own certificates. My Debian 11 uses two certificates for this, including Debian Secure Boot Signer 2021 and Debian Secure Boot CA. These certificates ensure that non-Microsoft signed code can load. A so-called “shim” bootloader is signed by Microsoft and will load the Debian-specific code and certificates. Therefore only the shim needs to be signed by Microsoft.

Also, in case this was not clear: UEFI Secure Boot requires an UEFI boot. If your OS was installed via the legacy BIOS system (you’re booting legacy), you can’t use UEFI features such as Secure Boot. You will need to migrate to UEFI first, which often involves a difficult bootloader swap or just reinstalling the entire system with UEFI enabled. Modern systems usually come with UEFI on by default.

My Odroid H2+ has two Realtek 8125B 2.5 Gbit network cards onboard. These are damn nice things, multi Gigabit for a cheap price. However, as with all new hardware on Linux, one question is always there: “Does it work with my Linux kernel?”.

If you run an operating system with Linux kernel 5.9 or higher (Debian 11, Ubuntu 20.04+ with HWE kernel…), the answer is yes: It will work out of the box, as the R8169 module/driver shipped with the kernel supports RTL8125B since 5.9.

However, my tests indicated that while the network cards indeed work, throughput wasn’t great: I was getting barely 2 Gigabit/s in an iperf3 test, nothing close to the 2.5 Gbit/s promised by the specification. So I had a look around whether there are better drivers. Indeed it looks like there is:

Realtek indeed seems to ship its own first party driver – it’s even open source and GPL licensed, but not included in the official Linux kernel source tree. Still, I tried out this project, which builds a nice DKMS kernel module out of the Realtek driver. DKMS is “Dynamic Kernel Module Support”, a framework to generate and load kernel modules easily. It takes care of things like recompiling your modules when you upgrade your kernel or switch configurations.

After installing the DKMS module, we had the new Realtek driver (module) available. However, even after a reboot it wasn’t active: My kernel seemed to prefer its inbuild R8169 module, so I had to explicitly block that (using the method described in the readme). After this, my kernel was forced to load my new Realtek R8125B module.

That was a great success! The new module achieved a steady 2.5 Gbit/s throughput with various HTTP/iperf speedtests, just like the specification promised.

Then I enabled UEFI secure boot. And all of my network connections broke 😞. I realized the issue: My DKMS module was not UEFI signed, so the Secure Process would refuse to load it. The OS would still boot, but without the module. Because I had disabled the R8169 module, no network card driver was available.

Obviously I could just have disabled secure boot again, but that would be no fun, would it? Instead, I wanted to get Secure Boot running and keep the R8125B Realtek module. After a bit of googling, it appears that the preferred way of doing this is by employing a so-called Machine-Owner Key (MOK). That is essentially just a certificate + private key that you generate, own, and control. You add that certificate to your UEFI certificate storage, where it will be retrieved by the shim loader. It will then be available to validate kernel modules. Note that this key can by default validate everything, including your own bootloaders, kernels etc. It is possible to limit the MOK to be only valid for kernel module validation (this involves setting a custom OID on the certificate), but doing this is out of scope for this blog post. Our certificate can be used for any purpose, including loading of DKMS modules – if you have signed them with your key.

There are various tutorials out there for various distributions and versions thereof. Here’s what I did – I hope that this is the recommended way, but I’m not entirely sure.

IMPORTANT: If you’re doing this on Debian (like I did), please read Debian’s notes about Secure Boot. It includes compatibility warnings and some pitfalls, such as validation issues in certain cases.

First of all, we need to generate our MOK, meaning a certificate and key. UEFI uses standard X.509 certificates, so if you’re familiar with them this is nothing new:

# Note: This command, and pretty much everything
# else needs to run as root.
openssl req -new -x509 \
-newkey rsa:2048 \
-keyout /root/uefi-secure-boot-mok.key \
-outform DER \
-out /root/uefi-secure-boot-mok.der \
-nodes -days 36500 \
-subj "/CN=Odroid DKMS Signing MOK UEFI Secure Boot"

For those unfamiliar with X.509/OpenSSL, here’s a quick overview: This command generates new RSA private key and corresponding self-signed certificate. We store the key in /root/uefi-secure-boot-mok.key (only accessible as root, important so that no malicious non-root party can use our key) and the certificate in /root/uefi-secure-boot-mok.der. The certificate is valid for 100 years, which isn’t ideal for security, but I am lazy. If you’re paranoid, you might want to lessen the lifetime and rotate your MOK from time to time. You might also want to consider using something else than RSA 2048 keys, but consider that your UEFI loading process must support it. The Subject Common Name (CN) is arbitrary, choose whatever your like.

Next, we need to import that certificate into our UEFI firmware store, where it can be loaded by the shim. Debian and other distributions (like Ubuntu) provide excellent tooling support for this: The Mokutils and MokManager. Basically, we just run this command (as root):

mokutil --import /root/uefi-secure-boot-mok.der

When you run this command, it will ask you for a password. Choose any password, but consider the following:

  1. You will need to type in this password in a short time again. You will only need this password once, it’s a one-time use thing.
  2. You will type in this password in an environment that will likely not use your native keyboard layout, but a default QWERTY one. If possible, choose passwords that you can type even on foreign keyboard layouts.

This does not actually do any real import. What this does instead is it marks this certificate as pending for inclusion for the MokManager. The MokManager is a EFI binary (i.e. a bootable system) included with UEFI-enabled Debian/Ubuntu systems. This is the actual workhorse that will do the import.

We now reboot the system, just use the reboot command or whatever you prefer. Ensure that you have access to a display and keyboard physically connected to your device. After the reboot, you will be presented by the MokManager waiting for you. You probably need to press some key to confirm, otherwise it will just revert to a standard boot. Once you are in the MokManager, just navigate through the options it presents to you. Press “Enroll MOK”, “continue”, “yes”, then enter the password we just setup. The MokManager may use a different keyboard layout! Once that is done and your password is accepted, select OK and wait for the reboot. Your MOK is now installed!

With the MOK installed, the final piece is to sign our DKMS modules with our key. This is very easy with Debian 11 installations, as there’s a ready made helper for this. We just need to edit two files:

Edit /etc/dkms/framework.conf and remove the comment regarding sign_tool:

## Script to sign modules during build, script is called with kernel version
## and module name
sign_tool="/etc/dkms/sign_helper.sh"

We also need to tweak the sign tool slightly, edit this file too and adjust the paths of the certificate and key:

/etc/dkms/sign_helper.sh
#!/bin/sh
/lib/modules/"$1"/build/scripts/sign-file sha512 /root/uefi-secure-boot-mok.key /root/uefi-secure-boot-mok.der "$2"

Just use the same filenames and path’s you’ve used while generating the cert + key above. This tutorial uses the same names and paths in all examples for consistency.

Now, we’ve almost done it! DKMS will now sign all modules while they’re build/installed, no additional configuration necessary!

In case you have already installed (unsigned) DKMS modules, we will need to re-build them to ensure they get signed. For my realtek module this involves the following (root again):

dpkg-reconfigure realtek-r8125-dkms

The exact command here varies depending on what modules you have, but the general idea is to call dpkg-reconfigure for each DKMS module you have. DKMS modules itself should be managed by Debian’s “.deb” package system, which is is called dpkg. The Realtek DKMS module is installed by a .deb package called realtek-r8125-dkms, hence the above command will re-install this DKMS module.

Finally, reboot a last time and your modules are now loaded – with UEFI secure boot on!

Categories
Server Administration Transport Layer Security (TLS)

Let’s Encrypt and expired root certificates

Let’s Encrypt is soon going to do something that I personally call “root-cross-signing”. This has definetly impact on some TLS clients, which is why I decided to give it some attention.

But first, let’s explain what the potential problem is: For a certificate to be trusted, it needs to be signed by a trusted Certificate Authority (CA) – the shift-of-trust model I already mentioned in earlier posts. Traditionally, the trust anchor – the point where the trust starts – used to be something called a root certificate1. Those are not signed by anyone (well, they technically sign themselves, but that doesn’t really do that much) and are shipped with your TLS client, or operating system (or similar). We call those certificates that are shipped with your client the trust store.

A root certificate usually signs at least one intermediate certificate (which in turn can sign other – lower – intermediates) and the lowest intermediate certificate signs the leaf certificate. That is the certificate of the actual service you’re connecting to, e.g this blog. This collection of certificates is what we call a certificate chain2. The server sends this chain to the client and the client then verifies this chain, checking whether all signatures are correct and whether the client trusts at least one (or the highest) of those certificates (remember, the clients has a trust store that contains root certificates)3. At least in theory. In practice this is much more difficult, as for example modern browsers to things like caching trust anchors + intermediates or auto-fetching intermediates, there is more than one chain, there is more than one root, chains are send with incorrect ordering and much more.

The actual issue

So, what has all of this to do with Let’s Encrypt?

As they tell us in their blog, they’re soon going to use a chain that looks like this: leaf -> R3 -> ISRG Root X1 -> DST Root CA X3. They’re going to use this chain for years to come. But DST Root CA X3 expires later this year! What happens then?

What will happen is heavily client specific, and depends on how your client verifies certificate chains. Some clients will not have a problem, others will completly break.

Let’s Encrypt tried to downplay the issue a bit, stating that Android compability is important and only “very old” other clients will have issues. I believe this isn’t the entire truth and I want to have a closer look at compatibility across clients, hence this blog post.

Why are we even doing this? Why not just go with ISRG Root X1?

As Let’s Encrypt explains, this is due to the fact that so many devices, especially Android, have older trust stores that do not get updates. Those devices have DST Root CA X3 in their trust store, but not ISRG Root X1. Hence the magic dance to try to support those old devices. But sadly, this will break compatibility with newer devices. Lets have a closer look at compatibility.

Compatibility across clients

Android

The new chain was specifically created for Android devices (>= 4.0) and thus no compatibility issues are expected on those Androids.

OpenSSL

OpenSSL has really bad chain validation in older versions and will thus try to validate up to the highest certificate in the chain, even if it should know that a lower certificate is a root and could just stop validating there. This means that older versions of OpenSSL will break with Let’s Encrypts new default chain, even if the trust store is up to date.

Affected is OpenSSL version less than 1.1 4. Yes, OpenSSL 1.1.0 was technically released 4 years ago, but many, many computers I know still ship something older. Take for example Ubuntu 16.04 which is supported until 2024 – that distro still uses OpenSSL 1.0.2g and thus needs the manual fix (see footnote above/below) in order to connect with Let’s Encrypt subscriber servers starting late 2021. Many people who need FIPS certification still use 1.0, because the FIPS module isn’t available in OpenSSL 1.1. I don’t have statistics about the used versions in the wild, but I expect millions of OpenSSL versions that aren’t yet running 1.1+ out here. Most of these will have trouble if no one does anything.

GnuTLS

A library I often see when I work with Debian/Ubuntu related things, or anything else that doesn’t like OpenSSL for some reason. Those products often tend to use GnuTLS, as its FOSS.

GnuTLS will break with the new Let’s Encrypt chain starting late 2021, unless you’re running a version newer or equal to 3.6.14 (released 2020). Again, lets have a look at what the distributions ship:

  • Debian 10 Buster currently ships GnuTLS 3.6.7, which is < 3.6.14, so it’s probably affected. Note that this is the latest Debian release, as of now there is nothing newer (but Debian Bullseye will be released this summer, which will have a fixed version). However, many people will run buster – or even older versions, you still see many stretch installations in the wild – for years to come. The GnuTLS versions here will probably all be affected. I’m not aware of a workaround for GnuTLS.
    Update: Numerous distributions have started to ship backports/workarounds to affected GnuTLS versions.
  • For Ubuntu, apparently everything older than Ubuntu 20.10 (Groovy Gorilla) is affected. This means that even the current LTS release Ubuntu 20.04 LTS “Focal Fossa”, supported until 2025+, is most likely affected (they apparently ship GnuTLS 3.6.13, one version below the fix).
    Update: Ubuntu recently backported patches to supported versions.

In summary, many distributions that don’t have a very fast version cycle will ship slightly older GnuTLS versions, and everything that is older than a year will likely break.

LibreSSL

Another library I sometimes see in the wild, especially when OpenBSD is involved. I’m not that familiar with it, so I’m going to make it short.

Everything less than 3.2.0 is affected, which was released in June 2020. So you need some version that is less than a year old to be not affected.

Windows, macOS and browsers native TLS libraries

Let’s Encrypt says those are not affected – at least someone who is fine.

Conclusion

I believe that Let’s Encrypts decision to use an Android-compatible chain by default will break setups for many people. That doesn’t mean that it was the wrong call, but it means that people need to be aware that their scripts, API integrations or whatever non-browser-non-android they have will most likely break.

What can I do if I don’t want all of this?

You can manually configure the alternate chain Let’s Encrypt will offer later this year. This means that you’re breaking Android compatiblity with Android < 7.1, but will preserve compatibility with all others, if their trust store contains ISRG Root X1 (which is a hidden requirement for all non-Android devices starting in late 2021, this applies for both chains).

Further reading

I highly recommend Ryan Sleevies Implementation Showdown, it explains in a much greater detail about how implementations did it wrong and it also gives you an idea what clients may be affected (it covers many more clients than those mentioned in this post).

Sources

Details about affected version numbers were taken from here:

Categories
Transport Layer Security (TLS)

Hidden openssl.cnf options and PrioritizeChaCha

Blarg, another long title. Again. Sorry.

So, this is something that I actually discovered a while (months) ago, so my memories are already a bit less fresh, but I think I still remember the important things.

I discovered this, because I had a common TLS CipherSuite config problem: I needed a server-site cipher order (e.g server has it’s own preferences), because I had some legacy weaker cipher enabled and we obviously don’t want clients with incorrect cipher order to connect with a potentially weak suite, when there are better suites available. But I also wanted to use ChaCha (more on that later).

If you don’t understand what I’m talking about, below is a summary on how TLS cipher suite negotiation works. If you know that stuff already, skip this chapter.

TLS cipher suite negotiation


TLS negotiates a cipher suite on each connection. There are many available – some are really secure, others are “okay-ish”, others are really bad. The really bad ones are usually disabled either at server or client-side (or both). But the “okay-ish” suites are generally enabled both at server and client side, even though client and server potentially also support the strong “good” ones.
What happens is that the client gives the server a list of cipher suites, which should be sorted in order of preference, and the server chooses one of them, depending on what is supported by both.

The server has generally two ways of choosing:

  • Server preference. The server maintains its own list of “preferred” cipher suites and chooses the best one on its list, depending on what the client supports.
  • Client preference. The server just chooses the first suite of the clients list that both parties support.

The thing with Client preference is, that there are (or were) some clients that send incorrectly ordered cipher suite lists – they have insecure ciphers at the top and more secure ones at the bottom. If the server lets the client choose, a weak cipher suite will be negotiated, even though both parties may support something stronger.

On the other hand, if the server maintains a correctly sorted list, one can guarantee that with server preference, the server will choose the most secure option depending on what the client supports. Thats the reason why server preference is the most common setting in real-world TLS.

Just use server cipher order and be done with it!

Yeah, that’s what many people do. And it was fine for some time. The thing is, TLS 1.2 at some point introduced a new cipher / cipher suite:

TLS_CHACHA20_POLY1305_SHA2561

That is an entirely new encryption algorithm (ChaCha with 20 rounds) and Poly1305 authentication. While those may not be exactly new, they were newly introduced into TLS.

What’s so special about ChaCha?

I won’t go into details here, I’m sure there are already posts elsewhere that cover this. Basically, ChaCha is (probably) secure and fast. This is interesting for machines that are not AES hardware accelerated (no AES-NI). This applies for example to most ARM based systems, like smartphones. Other examples may be IOT or embedded devices. Those don’t have accelerated AES and ChaCha is noticeably faster on those devices, especially when comparing against AES-256. Again, I won’t post big benchmarks here, those are elsewhere.

In summary, we would really want to use ChaCha on such devices.

Then just set ChaCha as preferred server cipher and be done with it!

But… I still like AES! Because that’s the thing: While ChaCha may be the new cool thing, many devices do have AES-NI (hardware accelerated AES) and thus can do AES faster than ChaCha. Plus, AES has been around for quite a while and so far we consider it secure. It’s also FIPS and similar certified, for people that are into that stuff.

So we still want to use AES! We just don’t want to use it with devices that have no AES-NI.

How can you get the best of both worlds?

What we need is something like this: If the client says “I don’t have AES-NI”, use ChaCha. If the client doesn’t speak ChaCha, or if it has AES-NI, use AES.

But how to determine if a client doesn’t have AES-NI? Fortunately, the standards made this not that hard: Modern clients that speak ChaCha will put ChaCha on the top of their cipher list, if they don’t have AES-NI.

Older clients, or clients with AES-NI will have something else on top. So the easiest way to use ChaCha with clients is to let the clients choose: If they have ChaCha on the top, use that. If not, use whatever else they have at the top. But that puts us back to the original problem above! What if an old, broken client puts legacy stuff on the top? There was a reason why we used server cipher preference.

Okay, so we want both client AND server cipher preference. That’s sadly something that OpenSSL can’t do. But OpenSSL developers saw the issue and offered us a solution:

OpenSSL has a flag for this!

Yep, that’s right. OpenSSL has a flag, called “PrioritizeChaCha” that does exactly what I described above: It will choose ChaCha if the client says “thats my most preferred cipher suite”, but will still honor the server cipher order in all other cases.

However, that is generally a compile-time flag. Meaning that you need to compile OpenSSL with this option if you want it. This isn’t really what I consider ideal. Many people use pre-compiled packages – after all, most distros ship programs this way. Especially when talking about security relevant stuff like OpenSSL, I personally like relying on the debian security team to update important packages. I just don’t favor the idea of compiling OpenSSL myself – it’s a lot more work for me, with not much benefits.

There’s another way: Setting this without re-compiling OpenSSL!

Yup, and that’s what this blog post was supposed to be about. The entire text wall above was just introduction for this.

OpenSSL has a config file. If you didn’t know that, I don’t blame you: I didn’t either, until last year or so. Since OpenSSL is mostly used as a library, there isn’t much to configure. On Debian-like systems, the file is /etc/ssl/openssl.cnf. The majority of this file is related to Certificate Authority settings. That’s because you can make your own CA with OpenSSL and that file is for persistently setting parameters needed for a (more or less) real CA.

But you can set more than just CA-specific parameters! The file can do more. Documentation has been sparse previously, but has been improved a lot recently. You can find documentation for it here. Thanks to Geert for pointing this out.

I will now try to give some guidance for that file. Ignore all the CA stuff – I’m not going to explain that. I’m interested in what this file can do to modify the TLS behavior.

Basically, the file is structured in sections, which are formatted like this: [section_name] followed by a list of settings valid for that section. A section ends when the next section starts.

In order to configure TLS options, we need to look at what sections are reponsible for that. The starting point is the setting openssl_conf which is part of some kind of “root section” that has no specific name. The default value of openssl_conf is set to default_conf. In order to define what default_conf should be, you can define a section called default_conf that sets everything you want.

Since we’re interested in TLS (SSL) specific settings, we set the ssl_conf parameter in our custom default_conf to the name of a section, let’s say ssl_sect (short for ssl_section). In the ssl_sect we then declare a setting system_default which holds some TLS options, unless the application overrides them explicitly. We set system_default to system_default_sect, again that is short for “system default section”.

In this section, we finally get to set our TLS/SSL parameters. You can for example define custom TLS min/max versions, default cipher suites and most importantly flags like PrioritizeChaCha.

[default_conf]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
MinProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=2
Options = PrioritizeChaCha,NoRenegotiation

The example above sets default parameters for each application that doesn’t specify anything else on it’s own:

  • Minimum TLS version is set to TLS 1.2
  • Some insecure cipher suites are disabled (security level is set to two). See the SECLEVEL documentation here and note the exact defintions of them tend to change between (major) OpenSSL releases.
  • Two flags are set: PrioritizeChaCha and NoRenegotation

The important things here are the flags: PrioritizeChaCha is explained in the beginning of this post. This config would enable the flag by default for all applications using OpenSSL. The other flag, NoRenegotation, disables TLS renegotiation. That’s mostly “for fun”. Renegotation was broken in the past, has not much use and was even killed in TLS 1.3, so not much reason to have it on in TLS 1.2 either.

Since I feel like this post is already way longer than intended, I will stop here. If you have any further questions about some of the topics covered here, feel free to use the comment section below. Or email me directly – my email is here somewhere.