Author: max

Categories
Linux Server Administration

Dedicated IP addresses and virtual machines

In today’s world, more and more things are running virtualized. Increasingly popular are those little things called “containers”. I feel like these are slowly replacing the “old” fully fledged virtual machines (VMs) in many areas. Yet they still exist and I still use them quite frequently.

The following talks mostly about my own typical server setup, which is Debian + VirtualBox. However, principles may apply to different setup types (non-Debian, containers) too.

When running a VM on a server, I often need to assign them dedicated IP addresses. How I do this depends a little on the host and the VM, but for my Debian + VirtualBox setups in the past I relied on a very old guide from Hetzner (partially still available here, german only). The guide pretty much suggested this config:

auto virbr1
iface virbr1 inet static
   address (Host IP)
   netmask 255.255.255.255
   bridge_ports none
   bridge_stp off
   bridge_fd 0
   pre-up brctl addbr virbr1
   up ip route add (Additional IPv4)/32 dev virbr1
   down ip route del (Additional IPv4)/32 dev virbr1

(This shows IPv4 only – IPv6 is highly similar, with inet6 instead of inet, all netmasks replaced by IPv6 compatible syntax and ip -6 instead of ip)

This is something that you would put into /etc/network/interfaces and then tell VirtualBox to use that interface as a bridge. Then you could configure the guest as you would configure a host by putting the additional IP as static IP and setting the host IP as gateway.

What this technically does is it creates a new interface using brctl (command from the bridge-utils package) which is then configured as some type of “fake bridge”, because we don’t actually assign it an interface to bridge to. Instead we tell the kernel that we want packages to our additional IP get forwarded into this virtual interface, where it gets picked up by our VM [This obviously requires forwarding enabled in the kernel, e.g net.ipv4.ip_forward=1 for IPv4 and net.ipv6.conf.all.forwarding=1 for IPv6].

This used to work nicely for quite a few years – I believe I’ve been using this setup since either Debian jessie or stretch – somewhere around that. However, on upgrading to Debian bullseye, it broke – the VMs would no longer receive any packets.

I’m still not sure what broke it – the new 5.10 kernel or a change in bridge-utils probably1 – but I found a solution, hence this blog post. Instead of creating a “fake bridge”, just use a tuntap virtual interface. My new workflow is like this:

Have a bashscript running on boot that pretty much does this:

#!/bin/bash
ip tuntap add mode tap virbr1
ip addr add <Host IP> dev virbr1
ip link set virbr1 up
ip [-6] route add <Dedicated IP>/<Netmask> dev virbr1

(I’ve retained the “virbr1” interface name from the example above for consistency)

You can probably convert the above bash script into a syntax compatible with /etc/network/interfaces, but I decided to not bother with that – nowadays theres often additional network management software installed which just interferes with the old file.

The approach is functionally still the same: It’s a routed configuration that forwards packets from the incoming physical interface to the virtual tuntap interface, where they get picked up by the VM – and vice versa for outbound packets. The use of the tuntap interface just avoids the bridged interface, which doesn’t work anymore anyway.

This approach seems to be suggested by the new Hetzner documentation, altough they lack examples on how to setup such a tap interface – hence my example above.

For full completeness, I will also shortly show how to configure a VM to use this virtual interface:

First of all, make sure IPv4/IPv6 packet forwarding is on – it’s not going to work otherwise. Second, configure VirtualBox to use the virtual interface as a “bridged adapter”, like this:

Screenshot from phpVirtualBox

If you don’t have a GUI for VirtualBox, you will need to figure out the VBoxManage command to do the same thing – good luck with that.

Then, configure your guest like this (example for /etc/network/interfaces)

auto enp0s8
iface enp0s8 inet[6] static
  address <Dedicated IP>
  netmask <Netmask>
  gateway <Host IP>

(The name of the interface – enp0s8 – depends on how your guest OS names the bridged adapter from VirtualBox – check ip a on the guest)

And that’s it. That’s the very short tutorial on how to assign your VM’s dedicated IP addresses (v4 or v6, or both).

Categories
Server Administration Transport Layer Security (TLS)

Let’s Encrypt and expired root certificates

Let’s Encrypt is soon going to do something that I personally call “root-cross-signing”. This has definetly impact on some TLS clients, which is why I decided to give it some attention.

But first, let’s explain what the potential problem is: For a certificate to be trusted, it needs to be signed by a trusted Certificate Authority (CA) – the shift-of-trust model I already mentioned in earlier posts. Traditionally, the trust anchor – the point where the trust starts – used to be something called a root certificate1. Those are not signed by anyone (well, they technically sign themselves, but that doesn’t really do that much) and are shipped with your TLS client, or operating system (or similar). We call those certificates that are shipped with your client the trust store.

A root certificate usually signs at least one intermediate certificate (which in turn can sign other – lower – intermediates) and the lowest intermediate certificate signs the leaf certificate. That is the certificate of the actual service you’re connecting to, e.g this blog. This collection of certificates is what we call a certificate chain2. The server sends this chain to the client and the client then verifies this chain, checking whether all signatures are correct and whether the client trusts at least one (or the highest) of those certificates (remember, the clients has a trust store that contains root certificates)3. At least in theory. In practice this is much more difficult, as for example modern browsers to things like caching trust anchors + intermediates or auto-fetching intermediates, there is more than one chain, there is more than one root, chains are send with incorrect ordering and much more.

The actual issue

So, what has all of this to do with Let’s Encrypt?

As they tell us in their blog, they’re soon going to use a chain that looks like this: leaf -> R3 -> ISRG Root X1 -> DST Root CA X3. They’re going to use this chain for years to come. But DST Root CA X3 expires later this year! What happens then?

What will happen is heavily client specific, and depends on how your client verifies certificate chains. Some clients will not have a problem, others will completly break.

Let’s Encrypt tried to downplay the issue a bit, stating that Android compability is important and only “very old” other clients will have issues. I believe this isn’t the entire truth and I want to have a closer look at compatibility across clients, hence this blog post.

Why are we even doing this? Why not just go with ISRG Root X1?

As Let’s Encrypt explains, this is due to the fact that so many devices, especially Android, have older trust stores that do not get updates. Those devices have DST Root CA X3 in their trust store, but not ISRG Root X1. Hence the magic dance to try to support those old devices. But sadly, this will break compatibility with newer devices. Lets have a closer look at compatibility.

Compatibility across clients

Android

The new chain was specifically created for Android devices (>= 4.0) and thus no compatibility issues are expected on those Androids.

OpenSSL

OpenSSL has really bad chain validation in older versions and will thus try to validate up to the highest certificate in the chain, even if it should know that a lower certificate is a root and could just stop validating there. This means that older versions of OpenSSL will break with Let’s Encrypts new default chain, even if the trust store is up to date.

Affected is OpenSSL version less than 1.1 4. Yes, OpenSSL 1.1.0 was technically released 4 years ago, but many, many computers I know still ship something older. Take for example Ubuntu 16.04 which is supported until 2024 – that distro still uses OpenSSL 1.0.2g and thus needs the manual fix (see footnote above/below) in order to connect with Let’s Encrypt subscriber servers starting late 2021. Many people who need FIPS certification still use 1.0, because the FIPS module isn’t available in OpenSSL 1.1. I don’t have statistics about the used versions in the wild, but I expect millions of OpenSSL versions that aren’t yet running 1.1+ out here. Most of these will have trouble if no one does anything.

GnuTLS

A library I often see when I work with Debian/Ubuntu related things, or anything else that doesn’t like OpenSSL for some reason. Those products often tend to use GnuTLS, as its FOSS.

GnuTLS will break with the new Let’s Encrypt chain starting late 2021, unless you’re running a version newer or equal to 3.6.14 (released 2020). Again, lets have a look at what the distributions ship:

  • Debian 10 Buster currently ships GnuTLS 3.6.7, which is < 3.6.14, so it’s probably affected. Note that this is the latest Debian release, as of now there is nothing newer (but Debian Bullseye will be released this summer, which will have a fixed version). However, many people will run buster – or even older versions, you still see many stretch installations in the wild – for years to come. The GnuTLS versions here will probably all be affected. I’m not aware of a workaround for GnuTLS.
    Update: Numerous distributions have started to ship backports/workarounds to affected GnuTLS versions.
  • For Ubuntu, apparently everything older than Ubuntu 20.10 (Groovy Gorilla) is affected. This means that even the current LTS release Ubuntu 20.04 LTS “Focal Fossa”, supported until 2025+, is most likely affected (they apparently ship GnuTLS 3.6.13, one version below the fix).
    Update: Ubuntu recently backported patches to supported versions.

In summary, many distributions that don’t have a very fast version cycle will ship slightly older GnuTLS versions, and everything that is older than a year will likely break.

LibreSSL

Another library I sometimes see in the wild, especially when OpenBSD is involved. I’m not that familiar with it, so I’m going to make it short.

Everything less than 3.2.0 is affected, which was released in June 2020. So you need some version that is less than a year old to be not affected.

Windows, macOS and browsers native TLS libraries

Let’s Encrypt says those are not affected – at least someone who is fine.

Conclusion

I believe that Let’s Encrypts decision to use an Android-compatible chain by default will break setups for many people. That doesn’t mean that it was the wrong call, but it means that people need to be aware that their scripts, API integrations or whatever non-browser-non-android they have will most likely break.

What can I do if I don’t want all of this?

You can manually configure the alternate chain Let’s Encrypt will offer later this year. This means that you’re breaking Android compatiblity with Android < 7.1, but will preserve compatibility with all others, if their trust store contains ISRG Root X1 (which is a hidden requirement for all non-Android devices starting in late 2021, this applies for both chains).

Further reading

I highly recommend Ryan Sleevies Implementation Showdown, it explains in a much greater detail about how implementations did it wrong and it also gives you an idea what clients may be affected (it covers many more clients than those mentioned in this post).

Sources

Details about affected version numbers were taken from here:

Categories
Server Administration Transport Layer Security (TLS)

Deploy CAA in exchange for internet cookies

Want do to something useful? Something that (slightly) improves your security in the internet? Want some internet cookies? Then please deploy CAA.

What is CAA?

CAA means Certificate Authority Authorization. In its essence, it’s just a DNS record. You know, just like those A records that resolve names into IP addresses, or MX records that list your mailserver(s). This DNS record tells other people which Certificate Authorities (CA) are allowed to issue certificates for your domain – nothing more, nothing less.

Why is that important? Who cares about this?

I do 🙂 . No seriously, it’s a useful thing, even though it may not seem like this on first glance. A CAA record on your domain improves the security of your domain. That is because by default every CA in the entire world can issue a certificate for your domain. That isn’t ideal, because there are more than 100 companies operating one or more root certificate authorities and much more subordinate Certificate Authorities, probably well over a thousand. Every one of these can in theory go rogue and forge a certificate for your domain(s). Are all of these thousand CA’s equally thrustworthy and secure? Well, from a browser security standpoint all of these are considered “trusted”, so everything signed by them will get a green lock in your browser (or any other application using TLS). Imagine just one of them getting hacked: Even if you are not a customer of them, they can still attack your website. You can detect such attacks using The Power Of Certificate Transparency, but preventing is better than detecting, right?1

With CAA, you can decide which CA you trust. If you only use a single CA, you can tell the entire world “I only trust them, no one else may issue a certificate for me”. That’s what CAA does. The DNS record prevents anyone else from issuing a certificate for your domain. Deploying CAA can be an important difference when a CA gets hacked or goes rogue for some reason.

How does this work on a technical side? How does a DNS record prevent someone from issuing? Can’t a rogue CA just ignore the DNS record?

Yes, they can in theory. However that would be really, really bad for them. CAA is mandatory for every single publicly trusted CA since 2017. No CA is allowed to simply “ignore” a CAA record2. It must respect the record – failing to do so might cause the CA to loose it’s publicy trusted status. CAA is like a law: Failing to follow the law will likely lead to punishment. That is why it’s important to deploy CAA: If no one deploys it, there’s no benefit in having it. It’s also possible to automatically monitor whether a certificate issuance was legal according to CAA (by using the power of Certificate Transparency and checking against the CAA record) and immediatly notify authorities if a mis-issuance happens.

Okay nice, now how do I go about deploying it?

First of all, we should talk about how CAA checking works from a CA point of view. The CA climbs the DNS domain names up to the TLD and checks each name for a valid CAA record. Let me explain it by example:

If we wanted to issue a certificate for blog.germancoding.com, the CA would first check for a CAA record at blog.germancoding.com. If no CAA record is found, the CA queries germancoding.com for a CAA record. Finally, if that was unsuccesfull too, the CA queries the TLD .com (yes, that’s technically also a domain3) for a record. If there isn’t anything either, the CA can consider that there isn’t any CAA record, so issuance is allowed by everyone (allow-by-default in this case). There are edge cases like when there’s a CName record involved, but this upwards climbing is the basic principle. CAA checking is stopped upon the first match, so a CAA record on blog.germancoding.com overrides one on germancoding.com.

What I’m trying to say here is that for most people a single CAA record on the main domain is enough – subdomains are covered automatically. Records on subdomains are only needed if you want to override something, for example:
“I only want Let’s Encrypt to issue for me, but on a single subdomain only I will additionally allow Sectigo” – this is easily doable due to the climbing-up principle explained above.

The actual record may look like this:

germancoding.com. IN CAA 0 issue "letsencrypt.org"

I think this format is called the Bind Zone Format, due to Bind’s popularity it’s seen everywhere (and due to the fact that it’s a standard). This means that there is a DNS record on germancoding.com which is of type CAA – CAA Records have their own type, which is kinda special these days – most protocols scramble data into DNS using a TXT record. The actual “data” of the Record is

0 issue “letsencrypt.org”

The first 0 is a flag. This flag is currently “reserved for future use” and should simply be set to zero (the actual definition of the flag is a bit more complicated, there’s already the “critical bit”, but let’s not blow up this post any more than neccessary).

The second part, “issue” describes what action we want to do (the definition calls this a “tag”): Currently defined tags are issue, issuewild, iodef, contactemail, contactphone. Most of these (iodef, contact*) are used for (real-time) reporting of CAA violations or issues. That isn’t really neccessary to implement, the issue/issuewild tags are the important ones.

The difference between issue and issuewild is that issue controls general issuance of certificates and issuewild controls the issuance of wildcard certificates. If no issuewild is present, issue takes over and also controls the issue of wildcards, so you do not need issuewild, unless you want special treatment for wildcard certificates.

The last part is the name of the certificate authority. The format here isn’t set in stone, each CA can name themselves, so you must check your CA documentation which name they use for their CAA checking. Most CA’s I’ve seen use their primary domain name, which is what is shown in my example (my example would be valid for Let’s Encrypt).

Once you have deployed this, you can use a TLS checker (like the one from Qualys) to see how it looks in the real world (you can also do a “dig CAA example.net” from a suitable box that has the dnsutils command). If you want to double check that everything works as intended, issue a certificate from your designated CA’s and see if you’re getting errors – if so, your CAA is too strict or incorrectly set and doesn’t allow issuance.

Excursion for specialists: The RFC 8657 CAA extension

The DNS CAA standard was recently (November 2019) expanded. The definition of what is new is documented in RFC 8657.

This extension is especially interesting for CA’s that use the ACME protocol (like Let’s Encrypt). With this extension, you have more fine-grained access control how certificates can be issued for your domain. For example if you’re familiar with the ACME protocol, you know that there are multiple challenges, e.g the DNS-01 and the HTTP-01 challenge (+ the forgotten ALPN challenge). With the extension from the RFC, you can define which challenges are allowed on your domains. You could, for example, only allow the DNS challenge. This would prevent someone who has the ability to re-reoute traffic to get certificates from Let’s Encrypt using the HTTP-01 challenge even when there’s a CAA record only allowing Let’s Encrypt. This obviously means that you must use the challenges allowed by your DNS CAA record yourself. An example of the extension in action can be seen on my main domain, germancoding.com:

# dig CAA germancoding.com

;; ANSWER SECTION:
germancoding.com.  3600  IN  CAA  0 issue "letsencrypt.org; validationmethods=dns-01"

As you can see, we can place a semicolon after the name of your allowed CA. After the semicolon we can set parameters, like the allowed challenges. The CA will disallow any challenge not listed in the CAA record, if the parameter is present.

There’s more than just the validationmethods. You can also bind to a specific account URI (e.g the account number or name you have at your CA), so that only accounts under your control can issue certificates – no one can create a new account at the same CA and use that for issuance, if you set that parameter.

Please note that this standard is too new to be supported everywhere. First of all, most CA’s do not use the ACME protocol and therefore things like validationmethods usually have no meaning to them. Secondly, even our ACME exemplary CA Let’s Encrypt doesn’t support the new extension yet in production. Nevertheless, you can still deploy it today on your domains, as support will be there in the future, I’m pretty sure – Let’s Encrypt already has it in staging and it’s only a matter of time until it’s deployed in production too.

Update: After many years, Let’s Encrypt enabled this feature in production in December 2022.