Revision as of 18:11, 5 August 2011

Debugging

Apache Segmentation Faults

If you see an error like this:

[Fri Feb 1 01:00:00 2011] [notice] child pid 12345 exit signal Segmentation fault (11)

Or

kernel: grsec: From 1.2.3.4: Segmentation fault occurred at 0000003000003dbc in /usr/sbin/httpd[httpd:15804]

You are experiencing a Segmentation Fault. A Segmentation Fault, also referred to as a segfault or a "bus error" occurs when the systems hardware notifies the Operating system that a memory access violation has occurred, and the OS then notifies the application (via a signal) about this condition. Most processes, by default, will then terminate the process and, if so configured, will also "dump core". One symptom of this with apache is a "blank screen" or an empty response. The child process or thread was killed, and it didnt finish whatever it was doing.

Lots of things can cause a sefault. For example, with Apache it could be a bug in Apache itself, one of its core elements (like APR), PHP, a web application, an external application or even a library. To find the cause of a segfault you need to generate a core file and perform what is called a backtrace. Disabling pieces of apache isn't a good way to find the cause. segfaults are memory errors, and simple "I disabled this and it stopped" cause and effect is extremely misleading. This article will provide some guidance on one method you can use to do that and to find the actual cause of the segfault.

We recommend you contact your OS vendor if you encounter any segmentation faults as this could be caused by any number of components in your Operating System itself.

Core Files

Core files allow you to diagnose what actually caused the memory fault on your system. Think of them as application logs, they will tell you exactly what was going on and when it stopped working correctly. To determine the cause, with a corer file, you must first configure your system to allow the creation of core files (this is usually disabled, as core files can use up a considerable amount of disk space).

Step 1:

Check that your system allows cores:

ulimit -c

If you see "0" that means they are disabled. You will need to set them to unlimited (keep in mind that if Apache is using more memory that you have room for our your disk you will eat up all your disk space, and you will get a core for EACH segfault, so dont leave this on for days, watch for cores, when you get a few turn off coredump support in Apache). To make core dumps unlimited, run this as root:

ulimit -c unlimited

Step 2:

Then you need to configure the application to "dump core" when it encounters a segmentation faulty. For example, if the segfaults are occurring with Apache, configure Apache to dump core:

Put the following in /etc/httpd/conf.d/debug.conf

 CoreDumpDirectory /tmp/apache2-gdb-dump

Make the directory as root, and check the permissions to make sure Apache can write to it

 mkdir /tmp/apache2-gdb-dump

 chown apache.apache /tmp/apache2-gdb-dump

 chmod og+rwx /tmp/apache2-gdb-dump

Step 3:

Install the debuginfo packages so your core file will contain the debug information needed to assist you in tracking down the source of the segfault:

 yum install httpd-debuginfo

If your OS is missing debuginfo, file a bug report with them. Although we do make debuginfo rpms available in the atomic repository for our Apache builds, unless you use our free Atomic rpms you will want to install the correct debuginfo file for your system.

Step 4:

Restart apache

 /etc/init.d/httpd restart

Backtracing

Wait for a segfault, and when you get one generate a backtrace:

gdb /path/to/httpd /path/to/core --batch --quiet -ex "thread apply all bt full" > backtrace.log

And look at the backtrace for the cause.

Keep in mind that memory use changes are not causes, so if you have a webapp or something that uses a lot of memory and you get segfaults unfortunately the memory use isnt the cause. Its just correlational, more memory in use means more opportunities for the fault to occur. It is not the cause. So if you have more memory in use and you segfault, and less and you don't thats not the cause of the problem. You can rule that out.

What changed?

If Apache was working correctly, and now you have an issue, always follow the golden rule "What changed?". One way to determine a potential cause of new errors is to check your systems update logs to see if any packages were update at the time the error began. This is not always the cause of the issue (or parts of the system may have changed, and this will only tell you about those changes that the package management system can log, not all the changes on the system):

Check your yum logs:

/var/log/yum.log

Ask yourself "When did the issue start, and what changed then?"

This will give you some idea if a module, OS, library, etc. update may be at the cause. Rollback and see if the error goes away, if they do, then you have a pretty good idea of what caused them.

If your rollback doesnt give you a root cause, then you need to determine:

• That you dont have a bug in Apache itself. Setup your system to capture core files, install the debuginfo for Apache and its modules and do a backtrace, then you'll see the cause with Apache
• That you dont have a bug in PHP, or some other major component in Apache (same advice applies in 1, get a core and backtrace)
• Check your webapps, you'd be surprised what they can do to themselves.
• Check your htaccess files to make sure you dont have a monster on your hands, a bad one can kill the server.

Please work with your OS vendor to rule out 1-4 before opening a support case. If you have a really good feeling that a component in ASL is causing an error, we would be happy to look into it, afterall those are provided by us and we support them. If you have a case where you can rule out your OS, PHP, Apache, etc., please make sure you have a core file for the error, such as a segfault and a full backtrace. We can't do anything without a core file and backtrace.