Atomicorp WAF Rules Troubleshooting
Specific Errors
Error creating rule: Could not add entry
This error means you are using an unsupported version of modsecurity, specifically 2.8.0. 2.8.0 has a critical bug in its IP matching code. It can not properly use IP addresses with rules, as documented in the github bug report below:
https://github.com/SpiderLabs/ModSecurity/issues/706
Do not use 2.8.0. Users should always use the currently support version of modsecurity documented at the URL below:
Atomicorp will not install 2.8.0.
Solution
Step 1) Remove 2.8.0 from your system
If you are using cpanel, disable modsecurity in cpanel and uninstall it (dont worry, the next step will show you how to install a bug free version)
For all other users, you will need to use the approved method for your distribution to remove 2.8.0. Please contact your OS vendor.
Step 2) Install aum on your system
aum will install modsecurity and keep both modsecurity and your rules up to date, automatically, with stable bug free versions that will never conflict with our rules. Please see the URL below for installation instructions for aum:
https://www.atomicorp.com/wiki/index.php/Downloading_Rules#Just_a_downloader
httpd: ModSecurity: WARNING Using transformations in SecDefaultAction is deprecated
This means that you are using out of date rules. If you are using Atomicorp rules, then this means you are not using the latest real time rules. The latest real time rules do not use transformations in the SecDefaultAction.
If you are using third party rules, this means that the rules contain a transform function in the SecDefaultAction setting. This is a deprecated setting.
ModSecurity: Failed to access DBM file "/var/asl/data/msa/
This means that you do not have the correct permissions setup for modsecurity to work correctly. Please make sure you have followed all the setup and installation instructions in the Atomic ModSecurity Rules wiki article. Specific guidance is provided in this section of the Setting Up Modsecurity guide:
Failed to create subdirectories
This means that the permissions on those subdirectories do not match the user that apache is running as. For example, if apache is running as the user "apache" those directories must be owned by apache.
Important note: If you are using an apache module that changes the user that apache runs as, on the fly, then those directories must be writable by all users. The use of modules that require you to set these directories as world writable and world readable is a serious security vulnerability, and should not be done. This will allow all users on the system to both change these logs, as well as to see both the logs and the payloads for all other users on the system.
Some modules, for example mod_ruid2, are not compatible with mod_security because they never write logs as anything other than the user for that instance, but will set permissions on subdirectories such that other users can not write or read from them. This is a known bug in mod_ruid2, and is not a bug in modsecurity. You will need to discuss this bug with the mod_ruid2 developers.
The most secure option is for these directories to be writable and readable only by a trusted user, and not by general users on the system.
Error creating rule: Unknown variable: MATCHED_VARS
Causes:
If you are getting this error, it means that you are using an old version of modsecurity that does not support the modern rule language and you are not running ASL. We recommend you install ASL.
ASL will not allow incompatible versions of the modsecurity rules to be installed. It will also automatically upgrade mod_security provided you have UPDATE_TYPE set to "all", which is the default. If you have updates disabled in ASL, then you will need to manually upgrade modsecurity:
yum upgrade mod_security
If you are not using ASL, then you will need to upgrade modsecurity using whatever method you used to install it. You will need to be running at least version 2.6.3.
Exec: Execution failed while reading output: /usr/bin/modsec-clamscan.pl (End of file found)
Without ASL
This error occurs if you install the 05_asl_scanner.conf file and/or the 99_asl_scanner.conf ruleset. These rulesets are not supported without ASL, as malware scanning is not included in the rules only subscription. ASL comes with malware upload scanning for HTTP, SSH, FTP and other protocols, including real time malware protection and much more. If you want malware upload protection, upgrade to ASL.
Solution:
Option 1. Update to ASL
Option 2. Remove the file 05_asl_scanner.conf and/or 99_asl_scanner.conf and restart apache.
With ASL
This means someone or something has disabled the malware protection system on your system. ASL can not and will not do this.
/usr/bin/modsec-clamscan.pl is not installed on the server.
Malware scanning is not included in the rules only subscription. ASL comes with malware upload scanning for HTTP, SSH, FTP and other protocols, including real time malware protection and much more. If you want malware upload protection, upgrade to ASL.
We also don't include that file or use the methods demonstrated in it because it doesn't scale very well. ASL has a binary streaming system that scales.
No action id present within the rule
This means that you using out of date rules. If you are using Atomicorp rules, then this means you are not using the latest real time rules. The latest real time rules have an id action for every rule.
httpd: ModSecurity: WARNING Using transformations in SecDefaultAction is deprecated
This means that you are using out of date rules. If you are using Atomicorp rules, then this means you are not using the latest real time rules. The latest real time rules do not use transformations in the SecDefaultAction.
If you are using third party rules, this means that the rules contain a transform function in the SecDefaultAction setting. This is a deprecated setting.
ModSecurity: Found another rule with the same id
This means that you have a modsecurity rule with the same id. All of our rules have unique id's. This error can only occur for one of two reasons:
1) You have loaded our rules twice. Please follow this installation guide and ensure that you have modsecurity configured as described in this document. These installation procedures will only load the rules once.
2) You have rules that are using id's already assigned to other rules.
If you are using third party or custom rules, check to make sure they have unique id's. Modsecurity requires a unique id for each rule. The range 300000-399999 is used by our rules, do not use this range for any custom rules, and if you have third party rules with these id's be sure to remove these rules.
Rule execution error - PCRE limits exceeded (-8): (null).
This is an internal limit to prevent a special type of DOS attack on the WAF itself. This is not caused by any of the rules. This is caused by the content the rules are inspecting. In certain cases, the content may be so complex that the WAF is stopping itself from doing too much work which could lead to a DOS attack on the system itself. If your system is generating these kinds of errors, it means you need to set the limits higher on your system, while it is beyond the scope of this article, another solution is to reduce the complexity of the content you are inspecting.
It is also possible this is occurring due to an actual DOS attack on your system. If you are certain this is not a DOS attack, simply increase these limits accordingly for your system. For example:
SecPcreMatchLimit 250000 SecPcreMatchLimitRecursion 250000
You may have to increase these limits for your system if you continue to get PCRE limit errors.
High memory usage
If apache is growing and growing in memory usage, and does not stablize please see the checklist below. If apache is simply using more memory, but has stablized (for example, Apache is using 100MB of memory, as opposed to 40MB of memory without modsecurity) this is normal. modsecurity caches the rules, and certain activities to speed up processing. Memory is infinitely faster than your hard drives, and loading the rules from the hard drive would be terrifyingly slow, too slow to be usable.
Ensure that modsecurity is setup correctly
An incorrectly configured modsecurity setup can cause this to happen. Some vendors that install modsecurity with their control panels do so incorrectly, and configure it in a manner that will lead to memory leaks with advanced security rules. The correct solution is to setup modsecurity correctly. There are two ways to do this:
Solution 1) Install ASL.
This will setup modsecurity correctly, and automatically.
Solution 2) Manually configure and setup modsecurity yourself.
Documentation for setting up modsecurity is available on the Atomic ModSecurity Rules page.
Ensure that modsecurity is built correctly
Solution 1) Install ASL
ASL will setup a correctly built version of modsecurity.
Solution 2) Install a modsecurity build from our rpm repositories.
You can download modsecurity here.
Solution 3) Install asl-lite
Please see the ASL Lite page for details. ASL Lite is unsupported software.
Check for leaking web applications
Web applications can and do leak memory. Check to make sure your memory leak isnt from a web application. Or a web application isnt amplifying a memory leak in something else.
Ensure that apache is built correctly
The major OS vendors, Redhat, Suse, Ubuntu and others do an excellent job of building Apache correctly. However, all software has bugs, and you should make sure that you are both running the latest updates from your vendor for a currently supported version of operating system from that vendor, and that you have ruled out an actual bug in apache. modsecurity is a complex piece of software, that relies of an even more complex piece of software, Apache, to work correctly. To improve performance, modsecurity will use apaches built in memory caching capabilities, and if there is a bug in these Apache supplied capabilities memory leaks can occur. We always recommend you discuss Apache memory leaks with your OS vendor first.
Please be especially diligent with source builds and Apache builds form non OS vendors. These builds are generally not as well tested, and so far are the only ones we have seen with memory management problems. If you use an Apache build from a non-OS vendor, you should discuss any memory management issues with them first if you have rules out the previous causes, as they are most likely a byproduct of a build error, or a bug in Apache.
Site loads slowly when the rules are loaded
There can be a number of reasons for this. Below is a list of the more common issues:
1. DNS based rules are activated, but there is no local DNS server on the system
Check to see if you have any DNS based rules loaded. Please see the mod_security page for a list of rules and which ones are DNS based.
These rules use DNS lookups to check for various characteristics of the connecting system, for example is it on a blacklist, is the IP address known to be used by a trusted source (Google for example). This means that for every connection to your system, a DNS lookup is done against the IP to see if the source IP is listed in some zone or blacklist. If you do not have a local DNS server, this can take a very long time depending on how busy the DNS server is that you are communicating with, and because the request is not cached, your system is asking the DNS server the same question several times in a row. This is because of the way HTTP works. A typical user request may need to open 100 or more connections to your site, and each time a DNS request is made to the RBL.
The simple fast solution is to make sure you are running a local DNS server on your system and that it caches requests. That way your RBL lookups are only to your local system, and if it already has the answer it will not have to ask the RBLs DNS server across the Internet. The difference in performance is night and day with a local DNS server, so we highly recommend you use on if you are going to use any RBL with any product.
2. The system is low on resources
As with any piece of software, the WAF will consume some resources on the system. If the system is short on resources the WAF has to compete with other parts of the system for those resources. Check to make sure you do not have any process that is using up RAM or cycles on the system.
3. Custom rules may be inefficient
If you are using custom rules, try disabling them. If a rule is written in an inefficient manner it can kill performance on the system.
4. Rules are loaded twice
If you have setup modsecurity yourself, check to make sure you arent loading the rules more than once. One several cpanel installations we've seen cases where users inadvertently setup modsecurity to load the same rules twice, and in one case a user set them up to load three times. Apache is a literal animal, if you load the same rule twice it will process that same rule twice, doubling the work on the system! And because our rules use advanced branching logic, if you load them multiple times that can have a very adverse effect on performance as the branching tree logic won't work correctly and you lose our on all the performance benefits a modern ruleset provides.
5. Suboptimal custom configuration
If you setup modsecurity yourself, your configuration could be suboptimal. You may have serial logging enabled, debugging enabled, pour inspection options selected, or any number of things could be wrong. We recommend you use our ASL product to setup modsecurity, its automatic and it will always configure the optimal settings for your system.
If you prefer to do it yourself, then please make sure you have read the modsecurity documentation and understand the options for modsecurity before you enable them. modsecurity is a powerful piece of software that can easily be misconfigured. Care should be taken when setting up this software yourself.
For do it yourselfers, we have put together a guide to help you on the Atomic ModSecurity Rules page.
