Richard Gunstone - Blog

Back to main page


Livepatching under Ubuntu and reflections on patch management (Featured)

So, how to get kernel livepatching for Ubuntu on your home devices? Indeed, what is kernel livepatching and it is worth using at all?

This, sometimes obscure feature, is a frequent feature of Cloud Service Provider VMs, but it's not that well known and can be easily overlooked.

First, a bit of history. Commonly mentioned in connection with Ubuntu Server (e.g. in CSPs), kernel live patching (KLP) is actually a feature of the Linux kernel. Some of the early workable implementations were based on the "Ksplice" project undertaken at MIT, later bought up by Oracl3. Live patches are distributed as kernel modules, which are dynamically loaded into the kernel at install time. Each module includes a complement of functions that are to take the place of existing functions being operated in the kernel. Once the livepatch is loaded and enabled, the kernel will start using any replacement functions automatically. The Linux kernel pages have a lot more information around the technical underpinnings of live patching, but in essence a variety of checks are carried out to ensure the process can operate smoothly. Once the new functions are active, existing uses of earlier functions are converged to using the new functions.

Livepatching has a number of limitations. First, only traceable functions can be patched. There are also limitations around the placement of the dynamic ftrace in relation to a function.

There is more information on the above available here, and Red Hat's presentation of the process is pretty good.

Livepatching support can be found in Amazon EC2 instances, on-prem Amazon Linux 2, Gentoo, Oracl3 Linux, RHEL 7 and later, SUSE, and Ubuntu 16.04 and later. Due to different implementations, the mechanism used in each distribution can vary, generally being based on livepatch, kpatch-git, kpatch, ksplice, or kGraft. Live patching is a desirable feature and is typically only available with a paid-for support package. If enterprise deployment is desired, then it's going to require a suitable budget and subscriptions to work.

Turning to Ubuntu, the kernel live patching capability is part of the distribution. Using the underlying capability for live patching, this enables the patch flow from Ubuntu to remediate critical vulnerabilities without the need for a system restart and re-load of the kernel. This is obviously extremely helpful in server scenarios, and where uptime must be maintained. Both high and critical vulnerabilities in the Linux kernel can be remediated in this way.

In Ubuntu, this is implemented using a client application (Livepatch) which connects to the Canonical live patch servers. In enterprise environments it's also possible to set up an on-premise server in place of the Cloud Canonical servers. Livepatches are cumulative, so later patches include the CVEs addressed in that and previous livepatches.

Livepatching in this scheme does not include CPU firmware and microcode, shared libraries and low-level dependencies, nor BIOS or EFI updates. If the kernel is hardened to prevent the loading of modules, which is the mechanism by which the likes of kpatch, kGraft and livepatch operate, then it may not be possible to perform KLP (in these cases a reboot will be necessary).

Once the livepatch is tested by Canonical, it is made available for release for the supported kernels

Unfortunately, livepatch is not available by default on Ubuntu. In some installations it is (namely CSP-provisioned VMs) but for the end-user further steps are necessary. This essentially boils down to creating an Ubuntu Advantage account. On first glance you might get the impression that UA then requires enterprise-level support packages to be purchased and used, but having a closer look in the UI reveals support for up to 3 hosts for free (personal use).

Once the UA account is created, you can copy and paste the key from UA into your root shell session, attaching the subscription and enabling the support as follows:

# ua attach [TOKEN]
# ua enable livepatch

It's worth noting that due the limited scope of livepatching under Ubuntu it's always a good idea to think about automatic updates (e.g. by using unattended-upgrades) or some other way of ensuring your hosts are patched regularly, as live patching is narrowly restricted to high and critical CVE's relating to the kernel and the production of live patches is vendor-driven.

You can find more information about livepatch under Ubuntu at the following link: Ubuntu Livepatch and Ubuntu Advantage.

So what, in conclusion, are the benefits and dis-benefits?

Patching is, essentially, a race against time, to ensure a host has the required patches quickly enough to reduce the potential for an attacker to develop a workable exploit (or, indeed, to take a workable exploit and use it to obtain unauthorised access to your systems). The time between a patch being made available, and the time a patch is applied, is effectively a window in which an attacker can (a) obtain the patch, (b) reverse engineer the patch to develop an exploit, and (c) to then use the exploit to attack a system. Notwithstanding this flow of events, the situation is obviously significantly worse if the attacker has harnessed a zero-day that has, unwittingly, been identified and remediated through the development of a patch by a supplier. The attacker, in the second scenario, is in posession of the exploit and could use it in advance of a patch being made available or during the window between patch availability and patch application. In either case, the pace at which a patch is applied has significant benefit and this is where live-patching can deliver ROI in reducing attack surface.

The shortcomings with live patching are mainly around change management - the risk of automated application of patches/changes of functionality that lead to faulty functionality or incompatibility with a managed configuration. While conventional patch cycles are typically managed, in an enterprise context, using an approved change management process, live-patching short-circuits this. In extremis, this can lead to catastrophic availability issues if a live-patch is applied that is incompatible with an existing change-managed enterprise configuration in one form or another. Related to this, and somewhat interwined, is the obvious risk of technical deficiencies in the live patch that then lead to availability issues or unexpected changes in functionality.

Ultimately live patching is a risk balance decision: accept the risk of automatic application of patches and the potential risk to availability as a reasonable mitigation in the effort to reduce the risk of high-impact exploitation, or not. In the final analysis after working through all the potential options, this usually boils down to a carefully managed approach that prioritises at-risk hosts and services over others where live patching may not be as essential (for example, hosts protected by other security controls such as gateways, reverse proxies, WAFs, API gateways, etc.)

On the risk presented by automatic mass-patching, Ubuntu at least has some options there if you have a server setup for livepatch on-prem on your LAN. See https://ubuntu.com/security/livepatch/docs/on_prem for more info.

The takeaways for a typical enterprise would therefore be:

June 30, 2022