Proxy

Introduction

If you use a proxy (including reverse proxies, UTM appliances, application layer firewalls, etc.) or CDN solution in front of an ASL or mod_security protected system, you must configure your web server to understand the actual source of the clients connection, and not the proxy or CDNs source IP. By default, web servers will correctly treat the CDN/proxies IP address as the source IP. Thats because the connection is coming from the CDN or proxy, and not from the actual client.

This will of course cause all sorts of problems with any WAF, and is something you must correct to use a WAF with your CDN/proxy correctly. Otherwise your WAF will block the proxy/CDN and not the actual attacker! Because there are so many different kinds of proxies and CDNs, this is something you will have to configure for your system and proxy/CDN of choice.

All reputable proxy and CDN vendors will be able to provide you with an official solution for using their proxy with your web server. Contact your proxy or CDN provider for support. The section below are provided as a courtesy to our customers to help them setup their systems, and to provide links to these vendors where they are known.

If you are unable to do this yourself, please contact us and we'll put a quote together for you from our professional services group.

Discussion

What is a proxy

Note: All CDNs use proxies.

What is a proxy? In short, its basically another web server, configured to make requests to another web server. Much like giving someone a "proxy vote", a proxy server requests the web page on behalf of the actual client. The actual client never connects to the actual web server, only the proxy does. So the web server is only ever communicating with the proxy, and will only see traffic from the proxy, as well as the IP address of the proxy, and not the client.

Proxy's and WAFs

When used with a WAF, like mod_security, a proxy will request traffic on behalf of the client. The client never communicates with the WAF. The WAF then inspects the request for attacks. For example, when the proxy passes traffic to apache, it will make the requests to apache using the proxies IP address, and not the actual clients IP address. Again, this is because the proxy is the system connecting to the WAF, not the client. This is where the term "proxy" comes from, its acting as a proxy for the actual client.

The WAF will then detect and report attacks as having come from the proxy, which is the actual source of the attack (its the proxy thats actually attacking the system). This is because the attack is in fact coming from the proxy, as far as your web server is concerned.

X-Forwarded-For headers

To address this, any decent proxy will pass on the actual source of the connection sent to the proxy server using a custom untrusted unverified header in the web request. And this is where the problems start from a security perspective. These proxy servers use what is referred to as "a forwarding header" to pass on this information, for example the X-Forwarded-For header. The format of this header differs from vendor to vendor, and some require special software be installed with your web server to make use of this header. There is no generic solution for this issue.

The biggest security problem is that this header can be forged, and therefore should never ever be blindly trusted. There is no way for your web server to know if this header is valid, or if the connection even came from a proxy.

All client side headers can in fact be forged. The key to using this header is to only trust specific sources that use these headers. In nearly ever case, that will be only be the proxy server, and you must restrict the use of this header otherwise. Do not blindly trusted "forwarding" headers. Attackers forge them all the time, and they can not be trusted. The only way to make "trusted" use of these headers is to only trust specific trusted hosts that send these headers, and to reject them for any other source. A better solution is to block all web traffic to your server except from your CDN or proxy server(s). For example, if you proxy traffic to apache, you should block any other traffic to apache so that it can only come from the proxy server.

To blindly trust these headers, you must configure your web server for the specific proxy solution you are using. Failure to do this will result in certain rule sets, such as the Crawler Protector and RBL rules, to not work correctly and if you blindly trust the forwarded-for headers you will be opening your system up to compromise.

Security Warnings when using Proxies

As previously mentioned, all headers sent from a client can be forged and should never be trusted. This includes the X-Forwarded-For header. Therefore if you do use a proxy/CDN solution we highly recommend you block all other traffic to your web server except from this proxy/CDN if you are blindly trusting client headers, for example the "forwarded for" headers proxies use. If you can not do this, then you will need to use a software add on to your web server that can restrict which hosts can send headers that will be blindly trusted, and which will not. This will allow you to trust these headers when your proxy/CDN sends them, but not any other source. Failure to do this can result in compromise of your server.

Secondly, improper use of a proxy can result in the proxy server being shunned by the active response system. It will also cause rules such as RBL rules, and the Crawler protector rules to not function properly (as the actual source of the connection will be reported by Apache as the proxy and not the actual clients IP). It is the users responsibility to work with their proxy vendor to properly configure their web server to report the actual source of the connection, and not the proxy servers IP. By default web servers will report the proxy servers IP, and not the clients IP, because thats whats actually connecting to the server. If you have not made any modifications to your web server, then it is not setup correctly to work with a proxy/CDN.

Each proxy vendor does this in a different way, and it is not possible to provide guidance for all potential proxy solutions. Please contact your proxy vendor for support with configuring your web server to report the actual source IP address, and not the proxy servers.

Note: Some proxy modules for apache do not handle IPv6 correctly. If you find that you have this problem with IPv6 enabled, please report this issue to your proxy module vendor. If you do not need to use IPv6, disabling IPv6 may resolve your problem, as some modules are known to break IPv4 as well when IPv6 is enabled (e.g. mod_rpaf).

Vendors Solutions

Note: The following information is provided as a courtesy to our customers. Contact your Proxy vendor for instructions about how to setup your web server with their proxy solution.

Specific Vendors

CloudFlare

Please contact CloudFlare for official support with their solution. This information is provied as a courtesy. You must configure your server when using Cloudflare to log the actual clients IP in your logs, and not the Cloudflare proxy servers IP. By default your web server will not do this. Failure to do so will result in shunning of the Cloudflare proxy servers, and will prevent rules such as the Crawler Protector and RBL rules from working correctly.

The following links are provided as a courtesy and in no way should be construed as supported by Atomicorp. Contact CloudFlare for instructions on setting up Apache to work with their solution.

Cloudflare generic guidance

http://www.cloudflare.com/resources-downloads

https://www.cloudflare.com/support

Cloudflare apache guidance

https://support.cloudflare.com/hc/en-us/articles/200170786#C5XWe97z77b3XZV

Generic Guidance

Note: This section is provided as a courtesy. While its not possible to cover every possible configuration, build and version of these web servers we have tried to provide complete guidance for the below web servers. If you are unable to setup your web server to work with a front end proxy please contact us and we'll have our professional services team put together a quote to get your custom environment working to meet your requirements.

Generic Nginx guidance

If you are using another proxy with nginx, this guidance may be helpful. You will need to ensure that you setup nginx correctly yourself for this condition, not all proxies work the same and this guidance may not work for your proxy solution.

nginx must be configured to report the actual IP of the connecting host. When nginx is placed behind a proxy it will only "see" the IP of the proxy in front of it, which will break a lot of things as the actual source of the connection will be hidden to nginx. For example, DNS based WAF rules will not work correctly, and shuns may result in shuns of your proxy server, this includes the Crawler Protector rules, RBL rules, and Spam URI rules amongst others. Shunning will also not work correctly as the proxy IP address will be detected as the source of the attack.

Therefore, to make this work correctly you will want to use a module with nginx that can automatically determine what the actual IP address is from the headers the proxy is passing to apache and will change the source information in apaches logs. The best place to find out what module(s) to use is from your proxy vendor. Some vendors require special proprietary modules to work with their proxy. You will also want to use a module that will automatically restrict the use of the "forwarded for" headers to a trusted set of sources, as discussed above, and to reject all other sources. Using a module like this restores nginx to its "normal" behavior, where nginx will report the actual source of the attack (and connection, which is helpful for log analysis tool) and not the proxies IP address(es).

The http_realip module is one example of a module that may help you to do this. Please check with your proxy vendor for assistance with the right tools to use, this module is neither provided by or supported by Atomicorp and may or may not provide what you need for your proxy solution. You can get the module from the URL below:

http://nginx.org/en/docs/http/ngx_http_realip_module.html

Instructions for its installation are available at the URL above. You will want to make sure that all of your nginx logs are logging the actual clients IP, and NOT the IP of the proxy server as the source.

Generic Apache Guidance

If you are using another proxy with Apache, this guidance may be helpful. You will need to ensure that you setup Apache correctly yourself for this condition, not all proxies work the same and this guidance may not work for your proxy solution.

Apache must be configured to report the actual IP of the connecting host. When Apache is placed behind a proxy it will only "see" the IP of the proxy in front of it, which will break a lot of things as the actual source of the connection will be hidden to Apache. For example, DNS based WAF rules will not work correctly, and shuns may result in shuns of your proxy server, this includes the Crawler Protector rules, RBL rules, and Spam URI rules amongst others. Shunning will also not work correctly as the proxy IP address will be detected as the source of the attack.

Therefore, to make this work correctly you will want to use a module with apache that can automatically determine what the actual IP address is from the headers the proxy is passing to apache and will change the source information in apaches logs. The best place to find out what module(s) to use is from your proxy vendor. Some vendors require special proprietary modules to work with their proxy. You will also want to use a module that will automatically restrict the use of the "forwarded for" headers to a trusted set of sources, as discussed above, and to reject all other sources. Using a module like this restores apache to its "normal" behavior, where apache will report the actual source of the attack (and connection, which is helpful for log analysis tool) and not the proxies IP address(es).

The mod_rpaf module is one example of a module that may help you to do this. Please check with your proxy vendor for assistance with the right tools to use, this module is neither provided by or supported by Atomicorp and may or may not provide what you need for your proxy solution. You can get the module from the URL below:

https://github.com/gnif/mod_rpaf

Instructions for its installation are available at the URL above. You will want to make sure that all of your apache logs are logging the actual clients IP, and NOT the IP of the proxy server as the source.

Here is a reference configuration for mod_rapf with Apache, again check with your proxy vendor as this reference is based on a local proxy that uses the referenced headers, and this configuration may not work with all proxies.

<IfModule mod_rpaf-2.0.c>
	RPAFenable On
	RPAFsethostname On
	RPAFproxy_ips 127.0.0.1
	RPAFheader X-Forwarded-For
</IfModule>

ipv6 and mod_rpaf

Important Note: some versions of mod_rpaf are known to have a bug that causes them to work incorrectly if you have the ipv6 protocol enabled on your system. Some versions of mod_rpaf provided with Plesk are known to have this bug. Other vendors builds may also have this bug.

This will occur even if ipv6 is not configured for an interface, but only if the ipv6 kernel module is loaded. Apache will detect that the system supports ipv6, if the kernel ipv6 module is loaded, and will pass this onto its modules. This bug in the third party mod_rpaf module, when ipv6 is enabled, will cause mod_rpaf to not pass on the actual FQDN for the IP address to apache. In those cases, rules that rely on this will fail to work correctly. If you have the ipv6 module loaded into your kernel, you will want to try unloading it if you find that mod_rpaf is not working correctly. If this resolves this issue for your system, please report this bug in the version of mod_rpaf you are using to your OS or control panel vendor.

You can disable ipv6 on boot by adding the following to /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.default.disable_ipv6 = 1

To disable in the running system:

echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6

echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6

or

sysctl -w net.ipv6.conf.all.disable_ipv6=1

sysctl -w net.ipv6.conf.default.disable_ipv6=1

You will need to restart apache and nginx once you have made this change.

Precautions

Layer 3 Firewalls

When traffic comes from a proxy, attacks also come from the proxy, itself. That is to say, the attacker isnt actually connecting to your system, the proxy is. This means that you cant block the actual attackers IP on your server with a layer 3 firewall, because the attackers IP never connects to your server only the proxy. With ASL, we try to help with this by allowing you apply your custom blacklists at layer 7 with this option:

http://wiki.atomicorp.com/wiki/index.php/ASL_WAF#MODSEC_00_BLACKLIST

Additionally, we support sending IP block requests to certain CDNs that support this capability. Please check with your CDN provider to see if they support this. Some providers limit the number of "firewall" rules you can implement on their CDN, its important to know what this limitation is as in some cases it may be too low for your needs.

FAQ

Why doesnt ASL automatically trust the Forwarded-For header?

Because its generated by the client, its trivial to forge and attackers send X-Forwarded-For headers all the time to trick poorly configured servers.

Blindly trusting these headers would make it trivially possible for any attacker to forge any source IP. Under no circumstances should you ever configure your system to blindly trust this header. No reputable WAF trusts this header blindly. This header should only be considered "trusted" if (1) its from a known trusted source and (2) you have configured your web server to only trust this header from those trusted sources and to reject/not trust this header from any other source.

This is what you will need to configure as described above to use this header in a trustworthy manner. Remember, this header is set by the client, in other words the attacker sets this header, not your web server, not the WAF. Your server has no idea if this header is valid, its up to you to configure your server to know what sources to trust, and what sources to not trust.

Why doesnt modsecurity automatically trust the Forwarded-For header?

See above.

Troubleshooting

HostnameLookups

If you are using any of the Crawler Protector rules (https://www.atomicorp.com/wiki/index.php/ASL_WAF#Crawler_Protector), then you must ensure apache is configured with this setting:

HostnameLookups Double

Some control panels are known to disable this setting, and doing so will cause Crawler Protector rules to fail to work correctly, causing them not to detect search engine crawlers correctly.

IPv6

See this article:

https://www.atomicorp.com/wiki/index.php/Proxy#ipv6_and_mod_rpaf