Blocking Attacks on Apache Behind a Load Balancer

OSSEC, a host-based intrusion detection system, comes with a lot of defenses set up and running right out of the box, including detecting malicious web requests and blocking them with firewall rules. This works really well for web servers not running behind load balancers, or when running OSSEC on the load balancers themselves, but doesn't work to actually block the attacks when your web servers are behind load balancers that aren't running OSSEC (for example, when using AWS ELB).

Fortunately, OSSEC has all the infrastructure needed to make it work — you just need to create a custom "active response" that updates your web server's configuration with the IP addresses to block. This is what we've done to make our Apache 2.4 servers block IP addresses as directed by OSSEC:

1. Log client IPs, not LB IPs

If you're running Apache 2.4 behind a load balancer, you should install mod_remoteip (there's also a backport of mod_remoteip for Apache 2.2 available that you can compile yourself). Make sure you register the header used by your load balancer to indicate the originating client IP address via the RemoteIPHeader directive, and the IP addresses (or blocks) used by your load balancer via the RemoteIPInternalProxy directive. Put these directives either in your base Apache config, or in your <VirtualHost> blocks. You'll also want to adjust any of the standard log formats you use (as well your own custom log formats) to replace %h with %a:

# name of the header from your load balancer
# that contains the originating client IP address
RemoteIPHeader X-Forwarded-For
# IP address or block of your load balancer
# (from the perspective of your back-end servers)
RemoteIPInternalProxy 10.0.0.1/24

# standard log file formats rewritten with %a in place of %h
LogFormat "%v:%p %a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%a %l %u %t \"%r\" %>s %O" common

After installing mod_remoteip and updating your Apache config, restart Apache and try accessing your server through the load balancer. You should now see your client IP address in the Apache access logs where your load balancer IP address used to be. This is a prerequisite for blocking web attacks with OSSEC — if your load balancer IP is in the logs instead of the originating client's IP, you'll end up blocking all access from your load balancer instead of just from the attacker!

2. Configure Apache to use a block list

Next, make sure you have mod_authz_core and mod_authz_host installed (which are usually installed by default) so that you can use them to block access to specific IP addresses in your Apache config. What we're going to do is have OSSEC manage a list of blocked IP addresses by writing them to a specific directory, one IP address to a file, in a format mod_authz_host interprets as denying access to the IP address.

If you've used earlier versions of Apache, you're probably used to access-control settings like this:

Order deny,allow
Deny from 1.2.3.4

In Apache 2.4, however, you instead use the Require directive for access control (optionally nested inside <RequireAll>, <RequireAny>, and <RequireNone> blocks to handle AND, OR, and NOT logic). So after upgrading Apache, you'd replace the above with this:

Require not ip 1.2.3.4

So for each IP address OSSEC wants to block, we'll have it write out Require not ip [the IP address] to a file, and keep all those block files in a specific directory. I chose /etc/apache2/authz for that directory, but it can be anywhere you want. We can source the files from that directory into our Apache config via the IncludeOptional directive.

So create that directory (as /etc/apache2/authz or your own preferred location), and then source it in beside your other auth directives in your Apache config. Since we're using negated auth directives in the sourced files (Require not ip 1.2.3.4), make sure you wrap all auth directives in your Apache config with a <RequireAll> block — outside of <RequireAll>, <RequireAny>, and <RequireNone> blocks, auth directives are combined using <RequireAny> semantics, rendering any negated auth directives moot.

Here is a simple example you might use in a <VirtualHost> block, allowing all GET, HEAD, and POST requests except those from the IP addresses included in our sourced directory:

DocumentRoot /srv/www
<Directory />
    # all included directives must pass
    <RequireAll>
        # allow only GET (and HEAD) and POST requests
        Require method GET POST
        # source additional directives from /etc/apache2/authz directory
        IncludeOptional authz/*.conf
    </RequireAll>
</Directory>

If you update your config with something like the above, and restart Apache, you should still be able to access your site as normal. But if you add a test.conf file to your /etc/apache2/authz directory with the following (replacing 1.2.3.4 with your own IP address), and restart Apache, Apache should now respond to you with a 403:

Require not ip 1.2.3.4

3. Create an OSSEC "active response" script

OSSEC comes with a few built-in active-response scripts, including the ability to add an offending IP address to the /etc/hosts.deny file, and to add the offending IP address to a firewall drop rule. It's easy to create custom active-response scripts, however; add this custom script as /var/ossec/active-response/bin/apache-deny.sh (or in the active-response/bin directory of wherever you installed OSSEC):

#!/bin/sh -e
# add/delete ip to/from apache authz deny list
# apache-deny.sh add - 1.2.3.4 1234567890.123456 1234

SCRIPT=$0
ACTION=$1
USER=$2
IP=$3
TIMESTAMP=$4
RULE=$5

# directory in which to add/delete authz entries
AUTHZ_DIR=/etc/apache2/authz
# assume /var/ossec/active-response/bin/apache-deny.sh
OSSEC_DIR=$(dirname $SCRIPT)/../..
# file to log activity
LOG_FILE=$OSSEC_DIR/logs/active-responses.log

log() {
    MSG="$(date) $SCRIPT $1"
    if [ "$ACTION" = "test" ]; then
        echo "$MSG"
    else
        echo "$MSG" >> $LOG_FILE
    fi
}

# log invocation of this script
log "$1 $2 $3 $4 $5"

# use python to validate legal ip address
IP_TYPE=$(cat <<EOF | python -
import socket
try:
    socket.inet_aton('$IP')
    print 'ipv4'
except socket.error:
    try:
        socket.inet_pton(socket.AF_INET6, '$IP')
        print 'ipv6'
    except socket.error:
        print ''
EOF
)

if [ "$IP_TYPE" = "" ]; then
    log "invalid ip $IP"
    exit 1
fi

# create safe name for authz deny file
AUTHZ_FILE="deny-$(echo -n "$IP" | tr -c '[:alnum:]' '-').conf"

case "$ACTION" in
    # add authz deny file
    add)
        echo "Require not ip $IP" > $AUTHZ_DIR/$AUTHZ_FILE
        service apache2 reload
        ;;
    # delete authz deny file
    delete)
        rm -f $AUTHZ_DIR/$AUTHZ_FILE
        service apache2 reload
        ;;
    # display test string
    test)
        echo "Require not ip $IP > $AUTHZ_DIR/$AUTHZ_FILE"
        ;;
    *)
        log "unknown action $ACTION"
        exit 1
esac

If you're using a directory other than /etc/apache2/authz for your block lists, replace the AUTHZ_DIR variable's value in the script with your custom directory path. Try it out by running the following command:

sudo /var/ossec/active-response/bin/apache-deny.sh add - 1.2.3.4

This should add a file named deny-1-2-3-4.conf to your /etc/apache2/authz directory with the following content:

Require not ip 1.2.3.4

The command also reloads your Apache config, so Apache will start acting on the newly added directive right way (you will need to adjust the script if your system has a different command for reloading Apache other than service apache2 reload). And if you run the command with the delete action, it will delete the same file:

sudo /var/ossec/active-response/bin/apache-deny.sh delete - 1.2.3.4

4. Configure OSSEC to invoke an active response

Now that Apache is configured to use the block list, and the apache-deny.sh script is in place to add IP addresses to and remove them from the list programmatically, we can configure OSSEC to trigger the apache-deny.sh script automatically whenever an existing OSSEC rule fires an alert at or above a certain level of importance. To do so, add two entries to your ossec.conf configuration file (typically located at /var/ossec/etc/ossec.conf).

The first entry registers our apache-deny.sh script as a command called apache-deny (and indicates it applies only to alerts that include a source IP). Insert it after the other <command> entries in your ossec.conf file:

  <command>
    <name>apache-deny</name>
    <executable>apache-deny.sh</executable>
    <expect>srcip</expect>
    <timeout_allowed>yes</timeout_allowed>
  </command>

The second entry configures OSSEC to execute the apache-deny command for all alerts at or above alert level 6. OSSEC has a bunch of built-in rules for general web attacks and general webapp security (as well as for common servers like Apache and Ngnix, and common platforms like WordPress, etc), so while you can tune the level setting to your own preferences, setting it to 6 will block anyone trying out common web-app vulnerabilities on your server. Insert it after the other <active-response> entries in your ossec.conf file:

  <active-response>
    <command>apache-deny</command>
    <location>local</location>
    <level>6</level>
    <timeout>600</timeout>
  </active-response>

The <timeout>600</timeout> setting above will direct OSSEC to use the same apache-deny command to remove an IP address from the block list after 600 seconds (10 minutes) has elapsed since adding the IP to the block list. You may also want to add a repeated_offenders entry to your ossec.conf, to extend the timeout for repeat offenders. Insert the following after the other <active-response> entries to extend the timeouts by 30 minutes, then 60 minutes, and finally 120 minutes:

<active-response><repeated_offenders>30,60,120</repeated_offenders></active-response>

One other thing you probably should configure in your ossec.conf, just to be safe, is to whitelist your load balancer IPs. Make sure you have a <white_list> entry in the <global> section of your ossec.conf that matches the RemoteIPInternalProxy setting you added to your Apache config in step 1:

<white_list>10.0.0.1/24</white_list>

Restart OSSEC, and then test it out by trying a URL for which OSSEC alerts automatically. Make sure you test from a box whose IP address is not in your OSSEC whitelist — otherwise you won't see OSSEC do anything. Here's an easy URL to test (replacing www.example.com with your load balancer's public-facing name):

curl -I 'https://www.example.com/?cmd.exe'

After trying that once (and giving OSSEC a second or two to do its thing), any further HTTP requests you make from that box for the next 10 minutes should result in a 403 error page from Apache. If that doesn't happen, look in OSSEC's main log file (/var/ossec/logs/ossec.log) and active-response log (/var/ossec/logs/active-responses.log) to check for errors executing the apache-deny active-response. If everything's working, there shouldn't be any messages about the apache-deny script in the main OSSEC log; but there should be an entry in the active-respone log that looks like the following:

Thu Jan  1 01:23:45 UTC 2015 /var/ossec/active-response/bin/apache-deny.sh - 1.2.3.4 1234567890.1234 31104

The last three fields in the log will be the IP address blocked, the unix timestamp, and the OSSEC rule that triggered the block.

5. Optionally add your own OSSEC rules

Simply using the out-of-the-box OSSEC rules is perfect for blocking all those run-of-the-mill bot drive-bys you see all the time from scanning your web site for common vulnerabilities. However, you can also add custom rules to direct OSSEC to detect potential attacks that are specific to your site, and either alert you, or block them automatically (or both).

One thing in particular you might do is add some rules that block simple/unintentional DOS attacks (especially if your site has certain pages that are especially vulnerable to DOS attacks, like with slow back-end queries, etc). Here is an example of a set of rules you might add to your /var/ossec/rules/local_rules.xml file to detect and block simple DOS attacks (in the form of an unusually large number of web requests from the same IP address in a short amount of time) site-wide, with more sensitive rules for more-sensitive URL paths:

  <rule id="900000" level="1">
    <if_sid>31100,31108,31101,31120</if_sid>
    <description>Any web request.</description>
  </rule>

  <rule id="900001" level="0">
    <if_sid>900000</if_sid>
    <url>^/css|^/img|^/js</url>
    <description>Static css/img/js request.</description>
  </rule>

  <rule id="900002" level="1">
    <if_sid>900000</if_sid>
    <url>^/foo/special|^/baz</url>
    <description>Special foo or baz request.</description>
  </rule>

  <rule id="900003" level="1">
    <if_sid>900000</if_sid>
    <url>^/foo|^/bar</url>
    <description>General foo or bar request.</description>
  </rule>

  <rule id="900010" level="6" frequency="10" timeframe="3">
    <if_matched_sid>900000</if_matched_sid>
    <same_source_ip />
    <options>alert_by_email</options>
    <description>Alert if more than 12 (10+2) requests from same IP in 3 seconds.</description>
  </rule>

  <rule id="900011" level="6" frequency="50" timeframe="30">
    <if_matched_sid>900000</if_matched_sid>
    <same_source_ip />
    <options>alert_by_email</options>
    <description>Alert if more than 52 (50+2) requests from same IP in 30 seconds.</description>
  </rule>

  <rule id="900012" level="6" frequency="5" timeframe="30">
    <if_matched_sid>900002</if_matched_sid>
    <same_source_ip />
    <options>alert_by_email</options>
    <description>Alert if more than 7 (5+2) special foo or baz requests from same IP in 30 seconds.</description>
  </rule>

  <rule id="900013" level="6" frequency="10" timeframe="10">
    <if_matched_sid>900003</if_matched_sid>
    <same_source_ip />
    <options>alert_by_email</options>
    <description>Alert if more than 12 (10+2) general foo or bar requests from same IP in 10 seconds.</description>
  </rule>

You'll have to customize the URL paths, frequencies, and timeframes (and add or substract rules as necessary) if you want to use something similar on your web site. Here's a rule-by-rule explanation of the above (keeping in mind that OSSEC rules work like a pipeline, where you have to connect later rules to earlier ones if you want the later ones to be able to operate on the events that earlier rules have already matched):

Rule 900000 identifies general web requests that haven't otherwise been handled by the default OSSEC rules (gathering the output of the 31100, 31108, 31101, and 31120 rules from the built-in /var/ossec/rules/web_rules.xml file that haven't been matched by any other built-in rules). The rules following 900000 will filter through all the requests that 900000 has collected:

  <rule id="900000" level="1">
    <if_sid>31100,31108,31101,31120</if_sid>
    <description>Any web request.</description>
  </rule>

Rule 900001 filters out any requests for URLs starting with /css, /js, and /img (note the <url> element does not handle full perl-style regexes — it instead uses a very limited string-matching syntax that recognizes only ^, $, and | as special characters). This will prevent any requests for our app's static CSS, JavaScript, or image files from being counted by the other rules that rely on rule 900000:

  <rule id="900001" level="0">
    <if_sid>900000</if_sid>
    <url>^/css|^/img|^/js</url>
    <description>Static css/img/js request.</description>
  </rule>

Rule 900002 identifies requests for our most-sensitive URLs (URLs starting with /foo/special or /baz). Rule 900012 (later on in the pipeline) will raise a higher-level alert if we get too many of these too quickly:

  <rule id="900002" level="1">
    <if_sid>900000</if_sid>
    <url>^/foo/special|^/baz</url>
    <description>Special foo or baz request.</description>
  </rule>

Rule 900003 identifies requests for our moderately-sensitive URLs (URLs starting with /foo or /bar). Since Rule 900002 was specified first, URLs starting with /foo/special will be filtered out by 900002; Rule 900003 won't match them, but will match all other URLs starting with /foo. Rule 900013 (later on in the pipeline) will raise a higher-level alert if we get too many of these too quickly:

  <rule id="900003" level="1">
    <if_sid>900000</if_sid>
    <url>^/foo|^/bar</url>
    <description>General foo or bar request.</description>
  </rule>

Rule 900010 raises a level-6 alert if we get 12 or more requests that have been matched by rule 900000 (not including those matched by rule 900001, 900002, or 900003) from the same IP address in under 3 seconds. Note that one of the eccentricities of OSSEC is that rules configured with the frequency attribute are fired only after the rule has been matched twice more than the configured value — so defining a rule with frequency="10" actually means that the rule must be matched 12 times before it fires.

Also, whenever you add new rules that will result in active responses (like this one will, assuming your apache-deny active response is also set at level 6), it's usually a good idea to include the alert_by_email option until your sure it's working smoothly — that way OSSEC will always send you an email whenever the rule fires, allowing you to check that it's firing only under the circumstances you want it to (and that it's triggered about as often as you'd expect). You can remove the alert_by_email option once your satisfied it's working as planned:

  <rule id="900010" level="6" frequency="10" timeframe="3">
    <if_matched_sid>900000</if_matched_sid>
    <same_source_ip />
    <options>alert_by_email</options>
    <description>Alert if more than 12 (10+2) requests from same IP in 3 seconds.</description>
  </rule>

Rule 900011 raises a level-6 alert if we get 52 or more requests that have been matched by rule 900000 from the same IP in under 30 seconds. So 900010 will result in blocking an IP address if we get a quick burst of requests from it (12 in 3 seconds), whereas 900011 will fire if we get a steady stream (52 in 30 seconds):

  <rule id="900011" level="6" frequency="50" timeframe="30">
    <if_matched_sid>900000</if_matched_sid>
    <same_source_ip />
    <options>alert_by_email</options>
    <description>Alert if more than 52 (50+2) requests from same IP in 30 seconds.</description>
  </rule>

Rule 900012 raises a level-6 alert if we get 7 or more requests matched by rule 900002 — our most sensitive URLs — from the same IP in under 30 seconds. So rule 900012 has a much lesser tolerance for requests than our general rules 900010 and 900011, blocking IPs after just a few requests:

  <rule id="900012" level="6" frequency="5" timeframe="30">
    <if_matched_sid>900002</if_matched_sid>
    <same_source_ip />
    <options>alert_by_email</options>
    <description>Alert if more than 7 (5+2) special foo or baz requests from same IP in 30 seconds.</description>
  </rule>

Rule 900013 raises a level-6 alert if we get 12 or more requests matched by rule 900003 — our moderately-sensitive URLs — from the same IP in under 10 seconds. It straddles the gap between the very-sensitive rule 900012, and the more tolerant rules 900010 and 900011:

  <rule id="900013" level="6" frequency="10" timeframe="10">
    <if_matched_sid>900003</if_matched_sid>
    <same_source_ip />
    <options>alert_by_email</options>
    <description>Alert if more than 12 (10+2) general foo or bar requests from same IP in 10 seconds.</description>
  </rule>

Keep in mind that whenever you do IP-based blocking like this, multiple users behind the same NAT (ie in the same office or using the same Internet connection) will appear to your servers as all having the same IP address — so be sure to allow for that when deciding for your site how many requests from the same IP should trigger alerts.