Revision as of 13:38, 29 June 2014

Introduction

If you use a proxy or CDN solution in front of an ASL or mod_security protected system, you must configure your web server to understand the actual source of the clients connection, and not the proxy or CDNs source IP. By default, web servers will correctly treat the CDN/proxies IP address and not the actual clients IP as the source IP. Thats because the connection is coming from the CDN or proxy, and not from the actual client.

This will of course cause all sorts of problems with any WAF, and is something you must correct to use a WAF with your CDN/proxy correctly. Otherwise your WAF will block the proxy/CDN and not the actual attacker! Because there are so many different kinds of proxies and CDNs, this is something you will have to configure for your system and proxy/CDN of choice.

All reputable proxy and CDN vendors will be able to provide you with an official solution for using their proxy with your web server. Contact your proxy or CDN provider for support. The section below are provided as a courtesy to our customers to help them setup their systems, and to provide links to these vendors where they are known.

Discussion

Note: All CDNs use proxies.

What is a proxy? In short, its another web server, configured to pass on requests to another web server. A proxy will then pass traffic to the WAF, which then inspects it for attacks. For example, when the proxy passes traffic to apache, it will make the requests to apache using the proxies IP address, and not the actual clients IP address. Again, this is because the proxy is the system connecting to the WAF, not the client. This is where the term "proxy" comes from, its acting as a proxy for the actual client.

The WAF will then detect and report attacks as having come from the proxy, which is the actual source of the attack (its the proxy thats actually attacking the system). This is because the attack is in fact coming from the proxy, as far as your web server is concerned.

To address this, any decent proxy will pass on the actual source of the connection sent to the proxy server using a custom header. And this is where the problems start. These proxy servers use what is referred to as "a forwarding header" to pass on this information. The format of this header differs from vendor to vendor, and some require special software be installed with Apache to make use of this header. There is no generic solution for this issue.

WARNING: This header can be forged, and should never ever be blindly trusted.

All proxy headers can in fact be forged, so the key to using this header is to only trust specific sources that use these headers. In nearly ever case, that will be the proxy servers, and you will want to restrict the use of this header otherwise. Do not blindly trusted "forwarding" headers. Attackers forge them all the time, and they are not trusted. The only way to make "trusted" use of these headers to only trust specific trusted hosts that send these headers, and to reject them for any other source. A better solution is to block all web traffic to your server except from your CDN or proxy server(s). For example, if you proxy traffic to apache, you should block any other traffic to apache so that it can only come from the proxy server.

You must configure your web server for the specific proxy solution you are using to do this for you. Failure to do this will result in certain rule sets, such as the Crawler Protector and RBL rules, from working correctly and if you blindly trust the forwarded-for headers you will be opening your system up to compromise.

Warnings

As previously mentioned, all headers sent from a client can be forged and should never be trusted. Therefore if you do use a proxy/CDN solution we highly recommend you block all other traffic to your web server except from this proxy/CDN if you are trusting client headers, for example the "forwarded for" headers proxies use. If you can not do this, then you will need to use a software add on to your web server that can restrict which hosts headers will be trusted, and which will not. This will allow you to trust these headers when your proxy/CDN sends them, but not any other source. Failure to do this can result in compromise of your server.

Secondly, improper use of a proxy cam result in the proxy server being shunned by the active response system. It will also cause rules such as RBL rules, and the Crawler protector rules to not function properly (as the actual source of the connection will be incorrectly reported by Apache as the proxy and not the actual clients IP). It is the users responsibility to work with their proxy vendor to properly configure their web server to log the actual source of the connection, and not the proxy servers IP in all logs. By default web servers will log the proxy servers IP, and not the clients IP. If you have not made any modifications to your web server, then it is not setup correctly to work with a proxy/CDN.

Each proxy vendor does this in a different way, and it is not possible to provide guidance for all potential proxy solutions. Please contact your proxy vendor for support with configuring your web server to log the actual source IP address, and not the proxy servers.

Note: Some proxy modules for apache do not handle IPv6 correctly. If you find that you have this problem with IPv6 enabled, please report this issue to your proxy module vendor.

Guidance

Note: The following information is provided as a courtesy to our customers. Contact your Proxy vendor for instructions about how to setup your web server with their proxy solution.

Specific Vendors

CloudFlare

Please contact CloudFlare for official support with their solution. This information is provied as a courtesy. You must configure your server when using Cloudflare to log the actual clients IP in your logs, and not the Cloudflare proxy servers IP. By default your web server will not do this. Failure to do so will result in shunning of the Cloudflare proxy servers, and will prevent rules such as the Crawler Protector and RBL rules from working correctly.

The following links are provided as a courtesy and in no way should be construed as supported by Atomicorp. Contact CloudFlare for instructions on setting up Apache to work with their solution.

Cloudflare generic guidance

http://www.cloudflare.com/resources-downloads

https://www.cloudflare.com/support

Cloudflare apache guidance

https://support.cloudflare.com/entries/22055786-How-do-I-restore-original-visitor-IP-to-Apache-Web-Servers-

Cloudflare IIS guidance

https://support.cloudflare.com/entries/22041122-How-do-I-correct-visitor-IP-with-Windows-IIS7-and-IIS8-

Cloudflare nginx guidance

https://support.cloudflare.com/entries/22051973-Does-CloudFlare-have-an-IP-module-for-Nginx-

Generic Guidance

Generic Nginx guidance

If you are using another proxy with nginx, this guidance may be helpful. You will need to ensure that you setup nginx correctly yourself for this condition, not all proxies work the same and this guidance may not work for your proxy solution.

nginx must be configured to report the actual IP of the connecting host. When nginx is placed behind a proxy it will only "see" the IP of the proxy in front of it, which will break a lot of things as the actual source of the connection will be hidden to nginx. For example, DNS based WAF rules will not work correctly, and shuns may result in shuns of your proxy server, this includes the Crawler Protector rules, RBL rules, and Spam URI rules amongst others. Shunning will also not work correctly as the proxy IP address will be detected as the source of the attack.

Therefore, to make this work correctly you will want to use a module with nginx that can automatically determine what the actual IP address is from the headers the proxy is passing to apache and will change the source information in apaches logs. The best place to find out what module(s) to use is from your proxy vendor. Some vendors require special proprietary modules to work with their proxy. You will also want to use a module that will automatically restrict the use of the "forwarded for" headers to a trusted set of sources, as discussed above, and to reject all other sources. Using a module like this restores nginx to its "normal" behavior, where nginx will report the actual source of the attack (and connection, which is helpful for log analysis tool) and not the proxies IP address(es).

The http_realip module is one example of a module that may help you to do this. Please check with your proxy vendor for assistance with the right tools to use, this module is neither provided by or supported by Atomicorp and may or may not provide what you need for your proxy solution. You can get the module from the URL below:

http://nginx.org/en/docs/http/ngx_http_realip_module.html

Instructions for its installation are available at the URL above. You will want to make sure that all of your nginx logs are logging the actual clients IP, and NOT the IP of the proxy server as the source.

Generic Apache Guidance

If you are using another proxy with Apache, this guidance may be helpful. You will need to ensure that you setup Apache correctly yourself for this condition, not all proxies work the same and this guidance may not work for your proxy solution.

Apache must be configured to report the actual IP of the connecting host. When Apache is placed behind a proxy it will only "see" the IP of the proxy in front of it, which will break a lot of things as the actual source of the connection will be hidden to Apache. For example, DNS based WAF rules will not work correctly, and shuns may result in shuns of your proxy server, this includes the Crawler Protector rules, RBL rules, and Spam URI rules amongst others. Shunning will also not work correctly as the proxy IP address will be detected as the source of the attack.

Therefore, to make this work correctly you will want to use a module with apache that can automatically determine what the actual IP address is from the headers the proxy is passing to apache and will change the source information in apaches logs. The best place to find out what module(s) to use is from your proxy vendor. Some vendors require special proprietary modules to work with their proxy. You will also want to use a module that will automatically restrict the use of the "forwarded for" headers to a trusted set of sources, as discussed above, and to reject all other sources. Using a module like this restores apache to its "normal" behavior, where apache will report the actual source of the attack (and connection, which is helpful for log analysis tool) and not the proxies IP address(es).

The mod_rpaf module is one example of a module that may help you to do this. Please check with your proxy vendor for assistance with the right tools to use, this module is neither provided by or supported by Atomicorp and may or may not provide what you need for your proxy solution. You can get the module from the URL below:

http://stderr.net/apache/rpaf/

Instructions for its installation are available at the URL above. You will want to make sure that all of your apache logs are logging the actual clients IP, and NOT the IP of the proxy server as the source.

Important Note: some versions of mod_rpaf are known to have a bug that causes them to work incorrectly if you have the ipv6 protocol enabled on your system. This will occur even if ipv6 is not configured for an interface, and only the ipv6 kernel module is loaded. This bug in the third party mod_rpaf module, when ipv6 is enabled, will cause mod_rpaf to not pass on the actual FQDN for the IP address to apache. In those cases, rules that rely on this will fail to work correctly. If you have the ipv6 module loaded into your kernel, you will want to try unloading it if you find that mod_rpaf is not working correctly. If this resolves this issue for your system, please report this bug in the version of mod_rpaf you are using to your OS or control panel vendor.

You can disable ipv6 on boot by adding the following to /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.default.disable_ipv6 = 1

To disable in the running system:

echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6

echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6

or

sysctl -w net.ipv6.conf.all.disable_ipv6=1

sysctl -w net.ipv6.conf.default.disable_ipv6=1

You will need to restart apache and nginx once you have made this change.