Richard Gunstone - Blog

The key ingredients for good patch management governance (Strategy)

What has made enterprise patch management tougher recently is how dynamic and dispersed computing assets are, as well as the sheer number of installed software components to patch. In addition, patch management processes and technology take different forms depending on the type of assets (e.g., OT, IoT, mobile, cloud, traditional IT, virtual machines, containers). The result is that many organizations are unable to keep up with patching. Patching often becomes primarily reactive (i.e., quickly deploy a patch when a severe vulnerability is being widely exploited) versus proactive (i.e., quickly deploy patches to correct many vulnerabilities before exploitation is likely to occur) (NIST SP 800-40r4 [4])

Patching is the oft-mentioned bane of any information security practice. A significant proportion of security results can be obtained by getting the basics right, and patch management is one of those basics. Many organisations fail to do it correctly, having at best a hap-hazard patching regime relying on individual technical expertise or, worst, no coordination of patch management at all.

Statements like "we adhere to Patch Tuesday release cycles" or "our Linux hosts are patched using an on-premise staging server" are usually indicators of further work being needed. But it can be a beneficial baseline nevertheless: even for organisations working at these relatively low levels of patch management maturity, they were able to avoid exploitation by the WannaCry attack [1,2,3] – so even some automation is beneficial.

As SANS note [1,2,3], patch management is one of the biggest security and compliance challenges for organizations to operate effectively; many large data breaches were successful precisely because patches weren’t applied, often for a critical security update.

The underlying data for patching is not a great backdrop: consistently since 2001, the number of vulnerabilities registered with CVSS shows a trend that is increasing over time [1,2,3]. Even worse, much of this growth is in the medium and high severity categories, which outpace growth in low-severity vulnerabilities, thus over the last 20 years as a crude inference the risk of exploitation due to missing patches has significantly increased. Granted, CVSS data is not an easy read across to patches, and the value of CVSS is often discussed (c.f. prioritised vulnerability ratings etc.), but as a broadly interpreted indicator it is valuable in the context of patch management.

To make matters worse, the time between discovery of a vulnerability in an application or operating system and the emergence of a viable exploit has drastically shrunk to a matter of hours in some cases. This presents in some respects a toxic combination, which, of course, threat actors are very willing to exploit.

Yet, patch management has not received anywhere near the same level of emphasis in organisational governance or technical journals, and potentially to an extent compliance/audit frameworks.

So how does it all go wrong for so many organisations? The dearth of industry reporting suggests organisations suffer from:

• A total absence of documented policies and standards for patching. This is unfortunately far too common.
• Poor reporting and/or metrication. If you don’t know how effective your patching governance is, you can’t improve it.
• A reliance on technical configuration to perform patching, without centralised coordination.
• Where policies are in place, insufficient depth, modelling and consideration of patching as a concept from multiple perspectives. (This is one of the most valuable exercises you can do.)
• Patch applications seldom or poorly metricated. An inability to measure patch coverage means a process operating in the absence of measurement.
• An absent or inadequately managed Information Security Management System (ISMS), with policies and standards under centralised management. (Though usually the least exciting aspect of information security governance, a good ISMS has a lot of balancing checks including regular review cycles and similar that, if done correctly, create a continuous improvement cycle.)

In addition to all of these, software vendors have been changing their patch release practices that if anything create added yet hidden risk for corporate security. For example, several vendors like Microsoft are now rolling up patches to prevent patch fragmentation and integration issues [1,2,3]. While ostensibly the practice of rolling up patches is a good thing, since they often contain cumulative patches when installed, they create additional change management considerations for IT functions, which makes the overall process of patch management potentially more complex.

What good looks like

One useful practice to embed in an organisation is a coherent, cogent patch management policy and associated technical standards. The extent to which organisations can achieve the latter will vary, but an effective policy is a good baseline and enables:

• A focal point for setting out, in a clear way, the policy requirements the organisation wants to achieve.
• A locus of control that can be returned to, updated, and refined as the organisational information security posture develops.
• In terms of its implementation, a patch management policy enables an organisation to cater, simultaneously for:
  • Routine or BAU patching, which can be catered for in normal patch deployment cycles.
  • Fast-paced patching, typically due to incident response and/or vendor critical patches being released out-of-band. (higher maturity posture)
  • The ability to measure the effectiveness of patching processes, and to obtain situational awareness of the patch state for an entire organisation and if necessary take corrective action. (higher maturity posture)

Good patch management policies contain:

• The stereotypical policy document content. This is nevertheless important, as it establishes the governance and review cycle, and many other attributes needed for a successful policy.
• A scope definition: to what assets, people, departments, and so forth, the policy applies to.
• A framework for metrication and its regular reporting.
• Linkages to other policies where relevant, such as vulnerability management.
• Roles and responsibilities.
• Process for risk acceptance around patch management exceptions, and the criteria and associated for any exceptions.
• How vendor ratings for patches should be interpreted.
• Based on a range of factors, how quickly patches should be applied. For example, application of highly rated patches within a certain timeframe. Cyber Essentials for example notes that all Critical and High CVSS scoring CVE's should be addressed via patching within 14 days of the patch being released.
• A baseline for patch application, such as vendor release cycles. Although this is the most fundamental feature it’s essential and the mechanism by which a significant quantity of patching is performed.
• A mechanism for capturing patches that are not applied, for any reason, and how approval is obtained (or otherwise).
• Mechanisms within the policy for its enforcement.
• A continuous review cycle, developed in such a way to bring about a continuous improvement loop. Second only to the implementation of a baseline process, the CRP is the other key essential ingredient.

Policy governance for out-of-band (OOB) patches or emergency patches is a further element that good patch management policies embed. This element is often missing in patch management policies and applies to critical patches that are released by vendors typically for vulnerabilities that are being actively exploited or pose too significant a risk to their customer base to leave to a regular release cadence.

For larger organisations, OOB patching provisions in patch management governance enable incident response teams and any security operations capabilities to reduce attack surface exposure quickly by directing the relevant teams to apply patches once released and following any necessary integration/compatibility tests. Yet more mature organisations can implement risk assessment processes around OOB patching to perform impact analysis and make recommendations.

Of course, all of this is just a small part of the wider range of considerations, processes and governance. NIST SP 800-40r4 has some useful suggestions and is well worth a read [4]. The application of patches also touches upon a myriad of other areas, including how you approach patching of systems that are not online all of the time. An effective reporting regime usually produces some interesting results, with the main one being data that shows how difficult it can be to patch the entirety of the IT estate, which raises the obvious question as to how that is managed.

More widely, the culture (or "putting it into the corporate DNA") argument is often mentioned in the context of patch management, which revolves around a series complementary and interlocking strategies performed in an organisation to achieve good practices that are consistently followed.

Achieving cultural change is difficult, but to get there some key governance practices need to be put into place. An effective patch management policy (potentially with associated minimum standards) is a necessary first step to improve organisational security maturity - or, even better, implement it as part of comprehensive ISMS if you have the resources to do that.

References

1. Agile Security Patching, Michael Hoehl, 2018. https://www.sans.org/white-papers/38410/

2. Patch Management and the need for Metrics, Ken MacLeod, https://www.sans.org/white-papers/1461/

3. A Practical Methodology for Implementing a Patch management Process, Daniel Voldal, 2003. https://www.sans.org/white-papers/1206/

4. NIST SP 800-40r4, https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-40r4.pdf