Difference between revisions of "Apache"

From Atomicorp Wiki
Jump to: navigation, search
m (Backtracing)
 
(14 intermediate revisions by one user not shown)
Line 1: Line 1:
= Apache Segmentation Faults =
+
= Debugging =
 +
 
 +
== Apache Segmentation Faults ==
  
 
If you see an error like this:
 
If you see an error like this:
Line 9: Line 11:
 
''kernel: grsec: From 1.2.3.4: Segmentation fault occurred at 0000003000003dbc in /usr/sbin/httpd[httpd:15804]''
 
''kernel: grsec: From 1.2.3.4: Segmentation fault occurred at 0000003000003dbc in /usr/sbin/httpd[httpd:15804]''
  
You are experiencing a Segmentation Fault.  A Segmentation Fault, also referred to as a segfault or a "bus error" occurs when the systems hardware notifies the Operating system that a memory access violation has occurred, and the OS then notifies the application (via a signal) about this condition.  Most processes, by default, will then terminate the process and, if so configured, will also "dump core".  
+
You are experiencing a Segmentation Fault.  This is a generic error in all Linux systems, and is not caused by [[ASL]].  A Segmentation Fault, also referred to as a segfault or a "bus error" occurs when the systems hardware notifies the Operating system that a memory access violation has occurred, and the OS then notifies the application (via a signal) about this condition.  Most processes, by default, will then terminate the process and, if so configured, will also "dump core". One symptom of this with apache is a "blank screen" or an empty response.  The child process or thread was killed, and it didnt finish whatever it was doing. 
  
Core files all you to diagnose what caused the memory fault on your systemTo determine the cause, you must first configure your system to allow core files.
+
Lots of things can cause a sefault.  For example, with Apache it could be a bug in Apache itself, one of its core elements (like APR), PHP, a web application, an external application or even a library.  To find the cause of a segfault you need to generate a core file and perform what is called a backtraceDisabling pieces of apache isn't a good way to find the cause.  segfaults are memory errors, and simple "I disabled this and it stopped" cause and effect is extremely misleading.  This article will provide some guidance on one method you can use to do that and to find the actual cause of the segfault.
  
Step 1:
+
We recommend you contact your OS vendor if you encounter any segmentation faults as this could be caused by any number of components in your Operating System itself.
 +
 
 +
== Core Files ==
 +
 
 +
Core files allow you to diagnose what '''actually''' caused the memory fault on your system.  Think of them as application logs, they will tell you exactly what was going on and when it stopped working correctly.  To determine the cause, with a corer file, you must first configure your system to allow the creation of core files (this is usually disabled, as core files can use up a considerable amount of disk space).
 +
 
 +
===Step 1===
  
 
Check that your system allows cores:
 
Check that your system allows cores:
Line 23: Line 31:
 
ulimit -c unlimited
 
ulimit -c unlimited
  
Step 2:
+
===Step 2===
  
If the segfaults are occuring with Apache, configure Apache to dump core
+
Then you need to configure the application to "dump core" when it encounters a segmentation faulty.  For example, if the segfaults are occurring with Apache, configure Apache to dump core:
  
 
Put the following in /etc/httpd/conf.d/debug.conf
 
Put the following in /etc/httpd/conf.d/debug.conf
Line 31: Line 39:
 
   CoreDumpDirectory /tmp/apache2-gdb-dump
 
   CoreDumpDirectory /tmp/apache2-gdb-dump
  
Make the dir
+
Make the directory as root, and check the permissions to make sure Apache can write to it
  
 
   mkdir /tmp/apache2-gdb-dump
 
   mkdir /tmp/apache2-gdb-dump
  
Step 3:
+
  chown apache.apache /tmp/apache2-gdb-dump
 +
 
 +
  chmod og+rwx /tmp/apache2-gdb-dump
 +
 
 +
===Step 3===
 +
 
 +
Install the debuginfo packages for the applications you want to backtrace.  This is so your core file will contain the debug information needed to assist you in tracking down the source of the segfault.  For example, if you are backtracing just apache, then you need to install its debuginfo packages.  If you are debugging something else, like PHP, you need to install its debug symbols and so on. 
 +
 
 +
Here is an example for a package managed apache install:
 +
 
 +
  yum install httpd-debuginfo
 +
 
 +
'''If your OS is missing a debuginfo package for the application you want to backtrace, file a bug report with them.''' 
 +
 
 +
Note: if you have a source built apache, such as with cpanel, you will need to rebuild apache with debugging symbols.  For cpanel,  the following may work, but contact cpanel for assistance:
 +
 
 +
1) set the appropriate CFLAGS
 +
 
 +
export CFLAGS="-g"\x{2028}
 +
 
 +
2) Run easyapache
  
Install the debuginfo packages so your core file will contain the debug information needed to assist you in tracking down the source of the segfault:
+
/scripts/easyapache --build
  
yum install httpd-debuginfo
+
If this does not work for you, please contact your apache vendor for assistance.
  
If your OS is missing debuginfo, file a bug report with themAlthough we do make debuginfo rpms available in the atomic repository for our Apache builds, unless you use our free Atomic rpms you will want to install the correct debuginfo file for your system.
+
Note:  If you have apache modules loaded, please check to make sure they have the debug symbols installed as wellContact your OS or control panel vendor for assistance if you are unsure.
  
Step 4:
+
===Step 4===
  
 
Restart apache
 
Restart apache
Line 49: Line 77:
 
   /etc/init.d/httpd restart
 
   /etc/init.d/httpd restart
  
Step 5:
+
== Backtracing ==
  
 
Wait for a segfault, and when you get one generate a backtrace:
 
Wait for a segfault, and when you get one generate a backtrace:
Line 55: Line 83:
 
gdb /path/to/httpd /path/to/core --batch --quiet -ex "thread apply all bt full" > backtrace.log  
 
gdb /path/to/httpd /path/to/core --batch --quiet -ex "thread apply all bt full" > backtrace.log  
  
And look at the backtrace for the cause.
+
And look at the backtrace for the cause.  Most often the cause is a bug in Apache, or one of its supporting libraries.  '''Please make sure you share your backtrace with your Apache vendor first so they can rule out if this an Apache bug.'''  So far, we've only see apache bugs that cause segfaults, so if you want your segfault resolved quickly contact your Apache vendor first.  They can tell you if the issue is with Apache, or with mod_security.  If the issue is with Apache, we can't help you fix that.
  
Keep in mind that memory use changes are not causes, so if you have a webapp or something that uses a lot of memory and you get segfaults unfortunately the memory use isnt the cause. Its just correlational, more memory in use means more opportunities for the fault to occur. It is not the cause. So if you have more memory in use and you segfault, and less and you don't thats not the cause of the problem. You can rule that out.
+
Keep in mind that memory use changes do not cause segfaults.  So if you have a webapp, for example, that uses a lot of memory and you get segfaults unfortunately the memory use isnt the cause. Its just correlational, not causational.  More memory in use means more opportunities for the fault to occur. It is not the cause, it just increases the probability not the possibility. So when you have more memory in use and you segfault, and less and you don't segfault - the higher use of memory is not the cause of the segfault. Think of that as a warning that you have something else wrong.  You can rule memory use out.
  
= What changed? =
+
== What changed? ==
  
 
If Apache was working correctly, and now you have an issue, always follow the golden rule "What changed?".  One way to determine a potential cause of new errors is to check your systems update logs to see if any packages were update at the time the error began.  This is not always the cause of the issue (or parts of the system may have changed, and this will only tell you about those changes that the package management system can log, not all the changes on the system):
 
If Apache was working correctly, and now you have an issue, always follow the golden rule "What changed?".  One way to determine a potential cause of new errors is to check your systems update logs to see if any packages were update at the time the error began.  This is not always the cause of the issue (or parts of the system may have changed, and this will only tell you about those changes that the package management system can log, not all the changes on the system):
Line 73: Line 101:
 
If your rollback doesnt give you a root cause, then you need to determine:
 
If your rollback doesnt give you a root cause, then you need to determine:
  
1. That you dont have a bug in Apache itself. Setup your system to capture core files, install the debuginfo for Apache and its modules and do a backtrace, then you'll see the cause with Apache
+
* That you dont have a bug in Apache itself. Setup your system to capture core files, install the debuginfo for Apache and its modules and do a backtrace, then you'll see the cause with Apache
2. That you dont have a bug in PHP, or some other major component in Apache (same advice applies in 1, get a core and backtrace)
+
* That you dont have a bug in PHP, or some other major component in Apache (same advice applies in 1, get a core and backtrace)
3. Check your webapps, you'd be surprised what they can do to themselves.
+
* Check your webapps, you'd be surprised what they can do to themselves.
4. Check your htaccess files to make sure you dont have a monster on your hands, a bad one can kill the server.
+
* Check your htaccess files to make sure you dont have a monster on your hands, a bad one can kill the server.
  
 
Please work with your OS vendor to rule out 1-4 before opening a support case. If you have a really good feeling that a component in ASL is causing an error, we would be happy to look into it, afterall those are provided by us and we support them. If you have a case where you can rule out your OS, PHP, Apache, etc., please make sure you have a core file for the error, such as a segfault and a full backtrace. We can't do anything without a core file and backtrace.
 
Please work with your OS vendor to rule out 1-4 before opening a support case. If you have a really good feeling that a component in ASL is causing an error, we would be happy to look into it, afterall those are provided by us and we support them. If you have a case where you can rule out your OS, PHP, Apache, etc., please make sure you have a core file for the error, such as a segfault and a full backtrace. We can't do anything without a core file and backtrace.

Latest revision as of 13:01, 10 April 2014

Contents

[edit] Debugging

[edit] Apache Segmentation Faults

If you see an error like this:

[Fri Feb 1 01:00:00 2011] [notice] child pid 12345 exit signal Segmentation fault (11)

Or

kernel: grsec: From 1.2.3.4: Segmentation fault occurred at 0000003000003dbc in /usr/sbin/httpd[httpd:15804]

You are experiencing a Segmentation Fault. This is a generic error in all Linux systems, and is not caused by ASL. A Segmentation Fault, also referred to as a segfault or a "bus error" occurs when the systems hardware notifies the Operating system that a memory access violation has occurred, and the OS then notifies the application (via a signal) about this condition. Most processes, by default, will then terminate the process and, if so configured, will also "dump core". One symptom of this with apache is a "blank screen" or an empty response. The child process or thread was killed, and it didnt finish whatever it was doing.

Lots of things can cause a sefault. For example, with Apache it could be a bug in Apache itself, one of its core elements (like APR), PHP, a web application, an external application or even a library. To find the cause of a segfault you need to generate a core file and perform what is called a backtrace. Disabling pieces of apache isn't a good way to find the cause. segfaults are memory errors, and simple "I disabled this and it stopped" cause and effect is extremely misleading. This article will provide some guidance on one method you can use to do that and to find the actual cause of the segfault.

We recommend you contact your OS vendor if you encounter any segmentation faults as this could be caused by any number of components in your Operating System itself.

[edit] Core Files

Core files allow you to diagnose what actually caused the memory fault on your system. Think of them as application logs, they will tell you exactly what was going on and when it stopped working correctly. To determine the cause, with a corer file, you must first configure your system to allow the creation of core files (this is usually disabled, as core files can use up a considerable amount of disk space).

[edit] Step 1

Check that your system allows cores:

ulimit -c

If you see "0" that means they are disabled. You will need to set them to unlimited (keep in mind that if Apache is using more memory that you have room for our your disk you will eat up all your disk space, and you will get a core for EACH segfault, so dont leave this on for days, watch for cores, when you get a few turn off coredump support in Apache). To make core dumps unlimited, run this as root:

ulimit -c unlimited

[edit] Step 2

Then you need to configure the application to "dump core" when it encounters a segmentation faulty. For example, if the segfaults are occurring with Apache, configure Apache to dump core:

Put the following in /etc/httpd/conf.d/debug.conf

 CoreDumpDirectory /tmp/apache2-gdb-dump

Make the directory as root, and check the permissions to make sure Apache can write to it

 mkdir /tmp/apache2-gdb-dump
 chown apache.apache /tmp/apache2-gdb-dump
 chmod og+rwx /tmp/apache2-gdb-dump

[edit] Step 3

Install the debuginfo packages for the applications you want to backtrace. This is so your core file will contain the debug information needed to assist you in tracking down the source of the segfault. For example, if you are backtracing just apache, then you need to install its debuginfo packages. If you are debugging something else, like PHP, you need to install its debug symbols and so on.

Here is an example for a package managed apache install:

 yum install httpd-debuginfo 

If your OS is missing a debuginfo package for the application you want to backtrace, file a bug report with them.

Note: if you have a source built apache, such as with cpanel, you will need to rebuild apache with debugging symbols. For cpanel, the following may work, but contact cpanel for assistance:

1) set the appropriate CFLAGS

export CFLAGS="-g"\x{2028}

2) Run easyapache

/scripts/easyapache --build

If this does not work for you, please contact your apache vendor for assistance.

Note: If you have apache modules loaded, please check to make sure they have the debug symbols installed as well. Contact your OS or control panel vendor for assistance if you are unsure.

[edit] Step 4

Restart apache

 /etc/init.d/httpd restart

[edit] Backtracing

Wait for a segfault, and when you get one generate a backtrace:

gdb /path/to/httpd /path/to/core --batch --quiet -ex "thread apply all bt full" > backtrace.log

And look at the backtrace for the cause. Most often the cause is a bug in Apache, or one of its supporting libraries. Please make sure you share your backtrace with your Apache vendor first so they can rule out if this an Apache bug. So far, we've only see apache bugs that cause segfaults, so if you want your segfault resolved quickly contact your Apache vendor first. They can tell you if the issue is with Apache, or with mod_security. If the issue is with Apache, we can't help you fix that.

Keep in mind that memory use changes do not cause segfaults. So if you have a webapp, for example, that uses a lot of memory and you get segfaults unfortunately the memory use isnt the cause. Its just correlational, not causational. More memory in use means more opportunities for the fault to occur. It is not the cause, it just increases the probability not the possibility. So when you have more memory in use and you segfault, and less and you don't segfault - the higher use of memory is not the cause of the segfault. Think of that as a warning that you have something else wrong. You can rule memory use out.

[edit] What changed?

If Apache was working correctly, and now you have an issue, always follow the golden rule "What changed?". One way to determine a potential cause of new errors is to check your systems update logs to see if any packages were update at the time the error began. This is not always the cause of the issue (or parts of the system may have changed, and this will only tell you about those changes that the package management system can log, not all the changes on the system):

Check your yum logs:

/var/log/yum.log

Ask yourself "When did the issue start, and what changed then?"

This will give you some idea if a module, OS, library, etc. update may be at the cause. Rollback and see if the error goes away, if they do, then you have a pretty good idea of what caused them.

If your rollback doesnt give you a root cause, then you need to determine:

  • That you dont have a bug in Apache itself. Setup your system to capture core files, install the debuginfo for Apache and its modules and do a backtrace, then you'll see the cause with Apache
  • That you dont have a bug in PHP, or some other major component in Apache (same advice applies in 1, get a core and backtrace)
  • Check your webapps, you'd be surprised what they can do to themselves.
  • Check your htaccess files to make sure you dont have a monster on your hands, a bad one can kill the server.

Please work with your OS vendor to rule out 1-4 before opening a support case. If you have a really good feeling that a component in ASL is causing an error, we would be happy to look into it, afterall those are provided by us and we support them. If you have a case where you can rule out your OS, PHP, Apache, etc., please make sure you have a core file for the error, such as a segfault and a full backtrace. We can't do anything without a core file and backtrace.

Personal tools