Detecting and/or mitigating rapid exfiltration of RDBMS data

In this post I wanted to explore some of the lesser used features of iptables, and showing how iptables can be combined with a dual-homed Linux server to act as a data rate control. This is a cheap alternative to using a dedicated firewall.

There are numerous examples of relational databases being compromised and the contents being found on the dark web. What facilitates this activity in part (apart from poor security in the first place) is the availability of always-on, 24×7 connectivity, often using high-capacity Internet links.

Correctly sizing resources in terms of information security is usually never done. Capacity is more often than not over-provisioned, be that network capacity to manage peaks in traffic, virtualisation resourcing in hypervisors, or inactive disk quota for multi-user servers. Most of the time this is not an issue, but overprovisioning of resources can be used as part of a sophisticated attack and should not be overlooked, or, at a minimum, can undermine availability objectives due to excessive consumption.

Turning to our RDBMS example, there are lots of ways of slicing and dicing how the security architecture surrounding these kinds of services is implemented. Similarly, there are lots of things that can be done at the DBMS level to implement good security management, including well-defined privileges, good alerting, regular auditing, as well as multi-level controls on the underlying platform.

But lets suppose that the server, through some means or another, gets compromised. This could, for instance, be an insider using a privileged account on the RDBMS. The threat actor identifies lots of useful data on the database, and wants to transfer it out of the organisation. What can be done in this case to at least slow down the exfiltration of data, to allow other security processes to kick in?

There are many methods using a network to support exfiltration, ranging from port knocking channels, FTP, DNS tunnelling, packet manipulation, P2P, IM, BGP, or manipulated certificates. What can be done is down to the security policy of firewalls and other devices further away in the network.

The common element to many of these will be the data itself – exfiltrating a large amount of information will inevitably generate a lot of traffic. In some cases, such as DNS TXT exfiltration, a great deal. For some database applications, this is highly unusual.

We can achieve some control over this, using a simple mediating server. Architecturally, the pattern looks in essence like the following diagram:

Here we have a red side, black side, and a dual-homed Linux server between both (colours have no particular significance here). This server is configured to masquerade traffic from the RDBMS, and to forward ports from black to the RDMBS server on the red side. For good measure, the RDBMS server has a HBFW, and the Linux server should have an iptables default deny policy on the INPUT chain for eth0.

There is little protection between red and black due to the presence of the Linux server. The server does not need to be compromised in order to attack the RDBMS server, as we have not implemented a true protocol break nor verification at this point. Traffic from black will pass right through to red without interruption, as long as a permitting iptables rule applies.

However, this simple pattern allows us to implement some useful security functionality on the Linux server. In order to implement our traffic rate regulation control, we can turn to iptables, and in particular the hashlimit module.

The hashlimit module uses hash buckets to specify a rate limit, and this can be applied on the basis of host groups (source and/or destination IP address) or on a port-by-port basis.

Hashlimit allows a data rate to be specified as bytes per time period, where the time period can be second, minute, hour or day. If traffic exceeds the specified limit, then the iptables rule is triggered. You can also specify a burst value, to allow traffic to burst for a period of time.

This module is a very relevant module for a number of security applications. For instance, limiting data transfer for ICMP ping packets could still allow occasional legitimate use but deny excessive use (or at least take some action that benefits security management).

So for our RDBMS application, we would want to limit the data rate between the internal red interface (eth0) to the black interface (eth1). We’d also want this to apply to forwarded traffic.

IPT=”/sbin/iptables”
${IPT} -A FORWARD -m hashlimit --hashlimit-above 100kb/sec --hashlimit-burst 1mb --hashlimit-mode srcip,dstip --hashlimit-name rdbmsthrot -i eth0 -j DROP

In this rule, packets that are forwarded and go above 100 kilobits per second (not kilobytes – kB), will be dropped.

What’s the impact of dropping packets in this way? In TCP the congestion window (cwnd) will be reduced due to packet loss, and re transmission will occur to account for packet loss. The overall effect will be a reduction in throughput from the sender, in this case the RDBMS server.

Alternatively, we could implement a log and drop chain to document the threshold breach. Something along the following lines would work:

LOGLVL=6
IPT=”/sbin/iptables”
${IPT} -N LOGNDROP
${IPT} -A LOGNDROP -j LOG --log-level ${LOGLVL}
${IPT} -A LOGNDROP -j DROP
${IPT} -A FORWARD -m hashlimit --hashlimit-above 100kb/sec --hashlimit-burst 1mb --hashlimit-mode srcip,dstip --hashlimit-name rdbmsthrot -i eth0 -j LOGNDROP

This provides us with a means of managing throughput, and we could feed any log entries produced to an upstream SIEM. If a legitimate requirement for greater throughput is needed, the rule can be updated through change management procedures.

Limitations include scope for a covert channel, as we have not implemented a protocol break in the Linux server. The hashlimit group is also defined as a source/destination pair, so exfiltration to multiple targets may still be possible.