Difference between revisions of "ASL Troubleshooting"
m (→Step 5: Check your email settings in ASL) |
m (→Step 8: Check to make sure a third party product has not changed ASL's permissions) |
||
| (41 intermediate revisions by 4 users not shown) | |||
| Line 123: | Line 123: | ||
/etc/init.d/tortixd start | /etc/init.d/tortixd start | ||
| − | '''If you are missing that file''', and you installed ASL | + | '''If you are missing that file''', and you installed ASL you may have told the installer not to install the GUI or someone may have removed this part of ASL from your system. Please re-run the ASL installer and reinstall ASL. The ASL installer is the best tool for installing ASL completely. Instructions for installing ASL are available on this page: [[ASL_installation]]. |
| Line 195: | Line 195: | ||
If you are using any third party firewall management tools, uninstall them and if you can not uninstall them disable them. You will also want to make sure you do not have any pre-existing firewall rules on your system. This is the most common cause of firewall related issues: pre-existing firewall rules. | If you are using any third party firewall management tools, uninstall them and if you can not uninstall them disable them. You will also want to make sure you do not have any pre-existing firewall rules on your system. This is the most common cause of firewall related issues: pre-existing firewall rules. | ||
| − | ==== Step 6a: Check to see if you have any firewall rules ==== | + | |
| + | |||
| + | ==== Step 6a: Check your tortixd ACLs ==== | ||
| + | |||
| + | If you have configured ASL to limit access to specific IPs, check this file to ensure your current IP(s) are on the list of custom user defined allowed IPs: | ||
| + | |||
| + | /etc/asl/firewall/tortixd-access-list | ||
| + | |||
| + | the format is one IP or CIDR per line, for example: | ||
| + | |||
| + | <pre> | ||
| + | 1.2.3.4 | ||
| + | 10.0.0.0/8 | ||
| + | </pre> | ||
| + | |||
| + | If your IP address is not included in this list you will need to add it to the file. Then run this command as root: | ||
| + | |||
| + | service asl-firewall restart | ||
| + | |||
| + | ==== Step 6b: Check to see if you have any firewall rules ==== | ||
Run this command as the root user (not via sudo): | Run this command as the root user (not via sudo): | ||
| Line 219: | Line 238: | ||
You have no firewall rules. The problem is not caused by the firewall on your server, continue to [https://www.atomicorp.com/wiki/index.php/ASL_Troubleshooting#Step_7:_Run_up_a_sniffer_to_make_sure_the_problem_is_not_upstream step 7]. | You have no firewall rules. The problem is not caused by the firewall on your server, continue to [https://www.atomicorp.com/wiki/index.php/ASL_Troubleshooting#Step_7:_Run_up_a_sniffer_to_make_sure_the_problem_is_not_upstream step 7]. | ||
| − | ==== Step | + | ==== Step 6c: Check to see if there are third-party firewall rules that are causing a conflict ==== |
By default ASL will generate firewall rules that look like this: | By default ASL will generate firewall rules that look like this: | ||
| Line 269: | Line 288: | ||
Please contact the developers of this software for assistance removing it. Once you've removed this product you will want to follow [https://www.atomicorp.com/wiki/index.php/ASL_Troubleshooting#Step_6d:_Reset_your_firewall_rules Step 6d] below to remove any saved rules from your system and reset your firewall to the ASL defaults. | Please contact the developers of this software for assistance removing it. Once you've removed this product you will want to follow [https://www.atomicorp.com/wiki/index.php/ASL_Troubleshooting#Step_6d:_Reset_your_firewall_rules Step 6d] below to remove any saved rules from your system and reset your firewall to the ASL defaults. | ||
| − | ==== Step | + | ==== Step 6d: Check to make sure you dont have your servers IP addresses blocked in your firewall ==== |
Note: If you are using a third party firewall tool, you will need to check with the developers of that application to see if its blocking your servers IP. These instructions will only tell you if ASL is currently blocking your servers IPs. | Note: If you are using a third party firewall tool, you will need to check with the developers of that application to see if its blocking your servers IP. These instructions will only tell you if ASL is currently blocking your servers IPs. | ||
| Line 290: | Line 309: | ||
| − | ==== Step | + | ==== Step 6e: Check to see if there are any ASL generated firewall rules that are causing a conflict ==== |
The quickest way to detetermine if your ASL firewall rules are causing an issue is to flush all your firewall rules: | The quickest way to detetermine if your ASL firewall rules are causing an issue is to flush all your firewall rules: | ||
| Line 321: | Line 340: | ||
'''If you still can not connect,''' the problem is not caused by the firewall rules on your server, continue to [https://www.atomicorp.com/wiki/index.php/ASL_Troubleshooting#Step_7:_Run_up_a_sniffer_to_make_sure_the_problem_is_not_upstream step 7]. | '''If you still can not connect,''' the problem is not caused by the firewall rules on your server, continue to [https://www.atomicorp.com/wiki/index.php/ASL_Troubleshooting#Step_7:_Run_up_a_sniffer_to_make_sure_the_problem_is_not_upstream step 7]. | ||
| − | ==== Step | + | ==== Step 6f: Reset your firewall rules ==== |
| − | If you have been making modifications to your firewall with the ASL firewall manager, you can reset your firewall | + | If you have been making modifications to your firewall with the ASL firewall manager, you can reset your firewall by following this process: |
| − | + | https://support.atomicorp.com/hc/en-us/articles/360000591068-How-to-reset-the-network-firewall | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
If you can now connect, then you had configured a firewall rule or rules that were blocking you from accessing the web console. Please see the [[ASL_firewall]] for documentation on how to use the ASL firewall manager. | If you can now connect, then you had configured a firewall rule or rules that were blocking you from accessing the web console. Please see the [[ASL_firewall]] for documentation on how to use the ASL firewall manager. | ||
| − | If you still can not connect, the servers firewall is not the cause of your issue, continue to step 7 | + | If you still can not connect, the servers firewall is not the cause of your issue, continue to step 7 |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
=== Step 7: Run up a sniffer to make sure the problem is not upstream === | === Step 7: Run up a sniffer to make sure the problem is not upstream === | ||
| Line 451: | Line 460: | ||
/var/asl/session | /var/asl/session | ||
| − | is owned by ' | + | is owned by 'tortix:apache' with permissions 770. |
The permissions and ownership should be: | The permissions and ownership should be: | ||
| − | ''drwxrwx--- 2 | + | ''drwxrwx--- 2 tortix apache'' |
=== Step 9: Check to make sure tortix is in the apache group === | === Step 9: Check to make sure tortix is in the apache group === | ||
| Line 516: | Line 525: | ||
aum -uf | aum -uf | ||
| − | |||
| − | |||
asl -s -f | asl -s -f | ||
| Line 591: | Line 598: | ||
example.com has no MX record'' | example.com has no MX record'' | ||
| − | ==== Step 5c: Check to ensure your server can resolve the mail servers IP address | + | ==== Step 5c: Check to ensure your server can resolve the mail servers IP address ==== |
Then check to make sure your server can resolve the mail servers IP address. For example, in the above the mail servers name is "mail.example.com". You will need to ensure that this resolves on your server, which this command: | Then check to make sure your server can resolve the mail servers IP address. For example, in the above the mail servers name is "mail.example.com". You will need to ensure that this resolves on your server, which this command: | ||
| Line 620: | Line 627: | ||
https://www.atomicorp.com/wiki/index.php/ASL_Configuration#OSSEC_MAX_MSG | https://www.atomicorp.com/wiki/index.php/ASL_Configuration#OSSEC_MAX_MSG | ||
| − | Is set to | + | Is set to a positive integer between 1 and 9999. This will allow ASL to send emails as they occur and at the amount per hour you have set in this field. If you put a number outside of this range in this field, it will cause the maild service to shut off and you will not receive email alerts. We recommend you keep this set to "1". Note: By default, only alerts that are a level 7 or higher are emailed. |
=== Step 7: Ensure that OSSEC is enabled === | === Step 7: Ensure that OSSEC is enabled === | ||
| Line 628: | Line 635: | ||
https://www.atomicorp.com/wiki/index.php/ASL_Configuration#OSSEC_ENABLED | https://www.atomicorp.com/wiki/index.php/ASL_Configuration#OSSEC_ENABLED | ||
| − | === Step 8: Ensure that OSSEC set to server mode === | + | === Step 8: Ensure that OSSEC is set to server mode === |
OSSEC will only send emails if it is set to "server" mode: | OSSEC will only send emails if it is set to "server" mode: | ||
| Line 658: | Line 665: | ||
</pre> | </pre> | ||
| − | If your mail sever is listening on that IP address you will see something similar to | + | If your mail sever is listening on that IP address you will see something similar to the example above. |
| − | If you do not see something similar to this, then your mail server is not running on that IP address, or you can not connect to that IP on that port. | + | If you do not see something similar to this, for example you see nothing, then '''your mail server is not running on that IP address, or you can not connect to that IP on that port''', move onto step 10. |
==== Step 9b ==== | ==== Step 9b ==== | ||
| Line 701: | Line 708: | ||
| − | You have no firewall rules. The problem is not caused by the firewall on your server, continue to [https://www.atomicorp.com/wiki/index.php/ASL_Troubleshooting# | + | You have no firewall rules. The problem is not caused by the firewall on your server, continue to [https://www.atomicorp.com/wiki/index.php/ASL_Troubleshooting#Step_11:_Run_up_a_sniffer_to_make_sure_the_problem_is_not_upstream step 11]. |
==== Step 10b: Check to see if there are third-party firewall rules that are causing a conflict ==== | ==== Step 10b: Check to see if there are third-party firewall rules that are causing a conflict ==== | ||
| Line 778: | Line 785: | ||
The quickest way to detetermine if your ASL firewall rules are causing an issue is to flush all your firewall rules: | The quickest way to detetermine if your ASL firewall rules are causing an issue is to flush all your firewall rules: | ||
| − | + | service asl-firewall stop | |
| − | If your firewall rules are flushed, you should see this | + | If your firewall rules are flushed, you should see this when you run the command "iptables -L -n". |
<pre> | <pre> | ||
| Line 795: | Line 802: | ||
If you see anything other than this, you have some third party firewall management product that has modified your firewall rules, or policy defaults. Please contact the developers of this software for assistance removing it, and restoring your firewall policy defaults. | If you see anything other than this, you have some third party firewall management product that has modified your firewall rules, or policy defaults. Please contact the developers of this software for assistance removing it, and restoring your firewall policy defaults. | ||
| − | '''If you are now able to connect,''' you had a firewall rule problem. If an ASL generated firewall rule caused this issue it will log this. Check your system logs for any ASL generated firewall rules, and your IP address. Run this command as root, replacing 1.2.3.4 with your IP address. | + | '''If you are now able to connect,''' (step 9), you had a firewall rule problem. If an ASL generated firewall rule caused this issue it will log this. Check your system logs for any ASL generated firewall rules, and your IP address. Run this command as root, replacing 1.2.3.4 with your IP address. |
grep ASL /var/log/messages | grep 1.2.3.4 | grep ASL /var/log/messages | grep 1.2.3.4 | ||
| Line 965: | Line 972: | ||
=== Step 12 === | === Step 12 === | ||
| − | If you dont see a packet exchange, the problem is with | + | If you dont see a packet exchange, the problem is with the mail server itself. Either your mail server does not accept connections, is not running a mail server, you have an upstream firewall that is blocking you (if you are using a remote mailserver) or you have a network problem that is preventing you from connecting to your mail server. Please contact your mail server vendor for assistance with connections issues with your mailserver. |
If everything is working, and you have confirmed that your mail server is listening, and you can successfully send a test message, then check your mail servers logs. This means that ASL is able to send email, and your mail server is just not sending it on your account. Please contact your mail server vendor for assistance with configuring your mail server to send email to your email account. | If everything is working, and you have confirmed that your mail server is listening, and you can successfully send a test message, then check your mail servers logs. This means that ASL is able to send email, and your mail server is just not sending it on your account. Please contact your mail server vendor for assistance with configuring your mail server to send email to your email account. | ||
If you require additional assistance with this issue, please send us the output of all these steps as we will need this information to assist you. Please do not contact us about issues concerning the correct configuration of your mail server. Please contact your mail server vendor for assistance with their products. | If you require additional assistance with this issue, please send us the output of all these steps as we will need this information to assist you. Please do not contact us about issues concerning the correct configuration of your mail server. Please contact your mail server vendor for assistance with their products. | ||
| + | |||
| + | ==== Step 12a ==== | ||
| + | |||
| + | This step is provided only as a courtesy. If your local mail server is not working, contact your mail server vendor for assistance. These instructions are provided without any support. | ||
| + | |||
| + | In some cases, your local mailserver (if you have configured ASL to use a local mailserver) may not be running on working. Restarting your local mailserver may resolve this issue. | ||
| + | |||
| + | ===== postfix ===== | ||
| + | |||
| + | To restart postfix run this command: | ||
| + | |||
| + | service postfix restart | ||
| + | |||
| + | ===== qmail ===== | ||
| + | |||
| + | To restart qmail, run this command: | ||
| + | |||
| + | service qmail restart | ||
== ASL web gui not running == | == ASL web gui not running == | ||
| Line 1,019: | Line 1,044: | ||
== No events in the ASL GUI == | == No events in the ASL GUI == | ||
| − | === Step 1: Test | + | === Step 1: Test ASL Web === |
| − | First check to make sure | + | First check to make sure that ASL Web is working. In the Recent Events tab of the Security Events window, set the level filter to 1 so you can see all events. By default ASL Web is set to a higher level to filter out lower importance events and to only display attacks and highly supicious events by default. If the ASL Web is showing lower level events, you may either not have any attacks or suspicious events to report, or you may have disabled or not installed all of ASL protection components. |
In that case, send a simple test from the local system, or another system that should be both blocked and logged. This article provides a test example: | In that case, send a simple test from the local system, or another system that should be both blocked and logged. This article provides a test example: | ||
| Line 1,027: | Line 1,052: | ||
[[Atomic_ModSecurity_Rules#Testing_to_see_if_the_rules_are_loaded]] | [[Atomic_ModSecurity_Rules#Testing_to_see_if_the_rules_are_loaded]] | ||
| − | === Step 2: Check | + | === Step 2: Check to make sure ASL is up to date === |
| − | If you | + | If you are running an older version of ASL, please see this article to [https://www.atomicorp.com/wiki/index.php/Upgrading_ASL upgrade]. |
| − | + | If you are running the current major revision of ASL, run these commands as root: | |
| − | + | */var/asl/bin/aum -u | |
| − | + | */var/asl/bin/asl -s -f | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | asl -s -f | + | |
| − | + | ||
| − | + | ||
| + | === Step 3: Verify that OSSEC is working correctly === | ||
Run this command as root: | Run this command as root: | ||
| Line 1,079: | Line 1,078: | ||
</pre> | </pre> | ||
| − | If everything is working correctly, you should see all of | + | If everything is working correctly, you should see all of the above processes running. |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
If any of these processes are not running, please try restarting ossec-hids by running this command as root: | If any of these processes are not running, please try restarting ossec-hids by running this command as root: | ||
| − | service ossec-hids restart | + | *service ossec-hids restart |
If OSSEC is not starting, or some of these services are not starting, please see this article for additional system checks to perform to find the root cause of this condition and solutions to apply to address them: | If OSSEC is not starting, or some of these services are not starting, please see this article for additional system checks to perform to find the root cause of this condition and solutions to apply to address them: | ||
| Line 1,097: | Line 1,088: | ||
https://www.atomicorp.com/wiki/index.php/ASL_error_messages#Command_executed:_.2Fsbin.2Fservice_ossec-hids_restart | https://www.atomicorp.com/wiki/index.php/ASL_error_messages#Command_executed:_.2Fsbin.2Fservice_ossec-hids_restart | ||
| − | |||
| − | + | === Step 4: Verify that mysql is working correctly === | |
| + | ==== Check for errors in the mysql log ==== | ||
| + | Some products, such as cpanel, configure mysql to not save logs. Please contact your database vendor if this is your case for advice on how to configure your database server to log errors as you will need to confirm the database is working correctly before moving on to the next step. | ||
| − | + | Mysql is normally configured to log errors to /var/log/mysqld.log. A corrupt database will result in an error in the mysql logs. A typical error may be one of the following (please check your logs for any mysql errors and not just the example below): | |
| − | + | *''Table './tortix/alert' is marked as crashed and should be repaired'' <br />If you see errors such as these, [https://www.atomicorp.com/wiki/index.php/ASL_error_messages#MYSQL_reports_a_table_is_.22is_marked_as_crashed_and_should_be_repaired.22 please see this faq article] for links to mysql procedures for fixing databases.<br /><br /> | |
| + | *''[Warning] Aborted connection #### to db: 'tortix' user: 'tortix' ... (Got an error reading communication packets)'' <br />This message typically indicates that the connection was closed after timing out. If you see errors like these, it may indicate that you have mysql configured with a low ''wait_timeout''. Restoring this setting to its default value of 28800 is recommended. | ||
| − | + | ==== Make sure mysql is listening on the IP and port you configured ==== | |
| + | A database that is not listening on a TCP port or the same port that you have configured ASL to use. Check to make sure your MySQL server is listening on port 3306 on the IP address you have configured ASL to use as its database server. | ||
| − | + | Additionally, a database server that is listening on a different IP address from the one configured for ASL can cause start up errors. Check to make sure your MySQL server is listening on port 3306 on the IP address you have configured ASL to use as its database server. | |
| − | + | Please see this setting: | |
| − | + | https://www.atomicorp.com/wiki/index.php/ASL_Configuration#OSSEC_DATABASE_SERVER | |
| − | + | And ensure you can connect to port 3306 on that configured IP or hostname. | |
| − | + | ==== Check to make sure the tortix mysql user permissions are correct ==== | |
| + | If you are receiving an error during database setup similar to this: | ||
| − | + | '''''Loading ASL Web database schema: ERROR 1227 (42000) at line 27: Access denied; you need the SUPER privilege for this operation''''' | |
| − | + | then you need to check the MySQL permissions. Specifically if the tortix mysql user is either unable to create or is unable to execute triggers, events will not show up properly. | |
| − | + | To resolve this, follow these steps: | |
| − | Login to mysql as the tortix user | + | *'''log in to mysql as the tortix user''' <br /> |
| + | **mysql -u `cat /etc/asl/config | grep OSSEC_DATABASE_USERNAME | awk -F\" '{print $2}'` -p`cat /etc/asl/config | grep OSSEC_DATABASE_PASSWORD | awk -F\" '{print $2}'` | ||
| + | **Once you are logged into mysql successfully, select the ASL database with this command:<br /><br />''use tortix;''<br /><br />Note: If you changed the default name for the ASL database, you will need to change this accordingly. | ||
| + | *'''verify that triggers have been created''' <br /> | ||
| + | **Type this command in the mysql prompt:<br /><br />''SHOW TRIGGERS;'' <br /><br />If no triggers are displayed, the tortix user does not have correct permissions. The following steps should correct the problem:<br /> | ||
| + | ***Login to mysql as the root / admin user | ||
| + | ***Type the following commands in the mysql prompt:<br /><br />GRANT ALL ON tortix.* TO 'tortix'@'localhost' IDENTIFED BY '[tortix user's password]';<br />GRANT SUPER ON *.* TO 'tortix'@'localhost' IDENTIFED BY '[tortix user's password]';<br /><br />Please note the following: | ||
| + | ****The tortix user's password may be found in ''/etc/asl/config'' under the variable ''OSSEC_DATABASE_PASSWORD'' | ||
| + | ****Check to see what IP mysql is bound to. If it is bound to an ip address other than 127.0.0.1, or mysql is skipping name resolution, then you will need to set the priviliges for the tortix user to use the IP address that mysql is bound to. Replace 'localhost' with the IP address in the above GRANT statements if this is the case. | ||
| − | |||
| − | + | === Run ASL Web validation === | |
| + | After any of the corrective actions above, or if none of the above resolved the problem, run the following command: | ||
| − | + | /var/asl/bin/asl --web_validate | |
| − | |||
| − | + | Data will be available in ASL Web on the first polling after correction, assuming any new events have come in, it does not require a refresh or relogging. Event data that existed prior to the correction will not be repaired. | |
| − | + | === Contact Support === | |
| + | If none of the above resolved the problem, please contact support and install our support keys as shown at the below link and we will have a look. | ||
| − | + | https://www.atomicorp.com/wiki/index.php/ASL_FAQ#How_can_I_give_atomicorp_support_access_to_my_system.3F | |
| − | == | + | == Firewall == |
| − | + | === FTP not working === | |
| − | + | ==== Background ==== | |
| − | + | FTP is inarguably the most complex protocol in use on the Internet today, if not of all time, it is also the oldest. Unlike other protocols, like SMTP and HTTP, FTP does not work over one port or even a predicable set of ports. FTP uses lots and lots of ports, and it dynamically selects the ports it uses both of the client and server side randomly. You might think FTP just works over port 21, but thats only the tip of the iceberg. FTPs actual "data" connections happen on random dynamically allocated ports that are unique for each client and connection! Which makes it rather difficult for '''any''' firewall to support it. To support FTP all firewalls require what are sometimes referred to as "helper" modules. These modules actually look for what appears to be FTP traffic on all those random, dynamic ports. And they have to do this passively, and in real time because the FTP server has no way to tell the firewall what ports it needs open. The firewall has to figure this out as it goes: dynamically opening and closing the ports that FTP needs to use when you are using any firewall. Without these modules FTP simply doesnt work. And, because FTP is unlike any other protocol in use on your system, thats why its important that not only your firewall rules be setup correctly for FTP, but that both your firewall and your operating system support it, understand it and manage the connections in real time correctly. The article below provides a brief overview of how FTP works: | |
| − | + | https://www.centos.org/docs/5/html/Virtual_Server_Administration/s2-ftp-proto-VSA.html | |
| − | + | ==== Solutions ==== | |
| − | + | ====If you are using the ASL kernel==== | |
| − | + | Step 0) Make sure you are not using a third party firewall | |
| − | + | This includes the "iptables" service and firewalld. Make sure you have removed and disabled any third party firewalls from your system. This guide is only for the use of the ASL firewall. If you are using a third party firewall, please contact the vendor for that firewall for assistance. | |
| − | + | https://www.atomicorp.com/wiki/index.php/ASL_prerequisites#firewalls | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | https://www.atomicorp.com/wiki/index.php/ | + | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
Step 1) Make sure you have port 21 open. | Step 1) Make sure you have port 21 open. | ||
| Line 1,203: | Line 1,190: | ||
If you still can not access FTP, check to ensure you are not using any third party firewalls, including upstream firewalls which would also be blocking this port, or do not support FTP. If you are using a third party firewall, please contact the vendor for that firewall for assistance. | If you still can not access FTP, check to ensure you are not using any third party firewalls, including upstream firewalls which would also be blocking this port, or do not support FTP. If you are using a third party firewall, please contact the vendor for that firewall for assistance. | ||
| − | + | ====If you are not using the ASL kernel==== | |
| + | |||
| + | Step 0) Make sure you are not using a third party firewall | ||
| + | |||
| + | This includes the "iptables" service. Make sure you have removed and disabled any third party firewalls from your system. This guide is only for the use of the ASL firewall. If you are using a third party firewall, please contact the vendor for that firewall for assistance. | ||
| + | |||
| + | https://www.atomicorp.com/wiki/index.php/ASL_prerequisites#firewalls | ||
| + | |||
| + | Step 1) check to make sure kernel FTP modules loaded | ||
Check to make sure your third party kernel includes the standard Linux FTP tracking modules. All Linux firewalls require special kernel modules to track FTPs data connections (FTP is a very complex protocol), without these modules no Linux firewall will work. These modules have been included with Linux kernels for over a decade, so your kernel should have these kernel modules. | Check to make sure your third party kernel includes the standard Linux FTP tracking modules. All Linux firewalls require special kernel modules to track FTPs data connections (FTP is a very complex protocol), without these modules no Linux firewall will work. These modules have been included with Linux kernels for over a decade, so your kernel should have these kernel modules. | ||
| Line 1,213: | Line 1,208: | ||
You should see these modules loaded: | You should see these modules loaded: | ||
| − | nf_nat_ftp | + | ''nf_nat_ftp |
| − | nf_conntrack_ftp | + | |
| + | nf_conntrack_ftp'' | ||
If they are not, you can load them with the standard Linux kernel module command: | If they are not, you can load them with the standard Linux kernel module command: | ||
| Line 1,221: | Line 1,217: | ||
modprobe nf_conntrack_ftp | modprobe nf_conntrack_ftp | ||
| − | If these | + | If you have a really old kernel you may not have these kernel modules, in which case you will need to load the older kernel modules |
| − | + | ''modprobe ip_conntrack_ftp | |
| − | + | modprobe ip_nat_ftp'' | |
| − | + | If your kernel is locked (that is it wont allow modules to be installed once the boot sequence is finished), please see this article: | |
| − | + | https://www.atomicorp.com/wiki/index.php/ASL_error_messages#modprobe:_FATAL:_Error_inserting_somemodule_.28.2Fpath.2Fto.2Fmodule.ko.29:_Operation_not_permitted | |
| − | + | If you are unsure about how to load kernel modules in Linux, on boot, please see this article: | |
| − | + | [[Installing_custom_kernel_modules_with_ASL]] | |
| − | If | + | If these commands do not work, please ask your kernel vendor to include the standard Linux FTP modules and for instructions for how to load them. All Linux firewalls require these modules to do support FTP with a firewall, or use the ASL kernel which will always load these standard Linux kernel firewall modules. |
| − | + | ====Step 3) If you are using encrypted FTP==== | |
| − | + | As explained above, FTP is a very complex protocol that works over multiple dynamic ports. When Linux is configured to use its kernel based firewall, it uses special "helper" modules to detect the dynamic FTP data connections. These connections do not occur over port 21 or 20, they are dynamic "high ports" (e.g. ports 1024 and higher). The client and server negotiate these (either the client sends the ports it wishes to use, or the server sends the port it wishes to use). When this occurs over an encrypted FTP session Linux can not see this negotiation happen (because its encrypted), and therefore will not dynamically open and close these high port data connections between the server and the client. Therefore, you can not use encrypted FTP with any firewall without permanently opening a range of ports on your firewall(s) for your FTP server(s) to use, and configuring your FTP server(s) to only use that range of ports (this is called "passive mode"), instead of dynamic picking ports (also called "active mode"). '''Please contact your FTP vendor for assistance with configuring your FTP server.''' Once you have correctly configured your FTP server, then you will need to permanently open these ports by adding them to your ASL firewall configuration. Please see the article below for the method thats appropriate for your needs: | |
| − | + | https://www.atomicorp.com/wiki/index.php/ASL_firewall | |
| − | + | Keep in mind that FTP was never designed to be encrypted, and in some cases when faced with multiple firewalls (including desktop firewalls) it may simply not be possible to pass the FTP data connections through successfully. For this reason, other protocols were designed to replace to FTP, such as SSH and HTTPS. | |
| − | + | ====== proftp example ====== | |
| − | asl - | + | Note: This is an unsupported example provided as a courtesy. Please contact your FTP vendor for assistance with configuring your FTP server. If you require our assistance, please contact support and we will put a quote together for professional services. |
| + | |||
| + | |||
| + | '''Step 1:''' | ||
| + | |||
| + | Edit /etc/proftpd.conf, and add the line below: | ||
| + | |||
| + | ''PassivePorts 60000 60500'' | ||
| + | |||
| + | '''Step 2:''' | ||
| + | |||
| + | Add in the passive ports range to your ASL firewall configuration. For example, if you are using fast mode you would add these ports in this format to your FW_INBOUND_TCP_SERVICES setting: | ||
| + | |||
| + | 60000:60050 | ||
| + | |||
| + | ''Note: proftps PassivePorts setting limits the amount of parallel connections and the number of connections (read directory listings and/or file transfers) per net.netfilter.nf_conntrack_tcp_timeout_time_wait seconds. If you will be transfering a lot of small files, you may need to open a wider range of ports. If you are not using the ASL kernel, you will need to contact your kernel vendor for assistance if this does not work for you.'' | ||
| + | |||
| + | '''Step 3:''' | ||
| + | |||
| + | Restart proftp | ||
| + | |||
| + | === Reseting to defaults === | ||
| + | |||
| + | To reset the firewall to only those rules generated by ASL, follow these steps: | ||
| + | |||
| + | # iptables-save > /etc/asl/firewall/backup.fw | ||
| + | # service asl-firewall stop | ||
| + | # rm -f /etc/asl/firewall/asl_stop.fw | ||
| + | # service asl-firewall start | ||
=== Third party firewall products === | === Third party firewall products === | ||
Latest revision as of 15:01, 29 January 2019
ASL Troubleshooting
[edit] Frequently Asked Questions
Please see the ASL FAQ page.
[edit] Specific issues
[edit] Install failed
Please review the entire output of the installer. If the installer encounters an issue it can not resolve it will report that. These are the most common errors users report to us for installation failures, and their solutions:
[edit] HTTP Error 401: Authorization Required
See this FAQ:
[edit] The requested URL returned error: 401
See this FAQ:
[edit] HTTP Error 401: Authorization Required Trying other mirror.
See this FAQ
[edit] HTTP request sent, awaiting response... 401 Authorization Required
See this FAQ
[edit] Error: Your Username/Password is incorrect, or your account has Expired.
See this FAQ [
[edit] Processing Conflict: mysql conflicts MySQL
This means that someone has replaced MySQL on your system with a third party unsupported version of MySQL. Please see the ASL pre-requisites for supported versions of MySQL:
https://www.atomicorp.com/wiki/index.php/ASL_prerequisites#Supported_versions
[edit] If you see none of these errors
If the ASL install failed, and none of these errors were reported, please send us the full output of the installer. We will need that information to assist you. If you have lost that information, please run the ASL installer again and send us the full output of the installer. It is safe to run the installer multiple times, and there is no need to uninstall anything.
[edit] Can't connect to Web GUI on port 30000
If you are unable to connect to the ASL GUI this can be caused by any number of factors, the service could be down, a firewall could be blocking the connection, an issue exists with the client, etc. To assist with the process of troubleshooting these we have put together this check list. Please run through each step.
Note: Run all of the commands in the steps below as the root user. Do not use sudo.
[edit] Step 1: Ensure that you have your system configured for ASL
Please make sure your system meets and is configured correctly for all the pre-requistes for ASL. These are documented on the ASL_prerequisites page.
[edit] Step 2: Check to make sure the ASL GUI is installed
Run this command as root:
rpm -qa | grep asl-web
If the ASL GUI is installed, you will see output similar to this:
[root@host ~]# rpm -qa | grep asl-web asl-web-3.0.25-3.el5.art
If the ASL GUI is not installed, you will see output similar to this:
[root@host ~]# rpm -qa | grep asl-web [root@host ~]#
To install the ASL gui, run this command as root:
wget -q -O - https://www.atomicorp.com/installers/asl |sh
This will re-run the installer. It is safe to run the installer over an existing installation. Please check the installer output for any error messages. Generally the web console is not installed either because it was not selected for installation, or because a core dependency for the web console is missing from your system. The most common problem is attempting to install ASL on systems with third party un-supported versions of mysql. Please be sure to check the Supported Versions of Mysql pre-requisites page if the installer is reporting errors with mysql packages.
[edit] Step 3: Check to make sure all ASL updates are installed
First, check to make sure a third party product has not excluded any ASL packages from installation or upgrade. You will want to check your /etc/yum.conf file to make sure it is not globally excluding things incorrectly. For example, some third party products that do not use software and package management will try to exclude the systems PHP packages from install/upgrade by incorrectly using a regular expression such as "php*" or "*php*". Disable all excludes.
Run these commands as root:
aum -u
yum -y upgrade --disableexcludes=all asl asl-web ossec-hids asl-php*
asl -s -f
[edit] Step 4: Check that the GUI service is running
As root, run this command:
ps auxwww | grep tortixd
If the service is running, you should any output similar to this (details will vary for your system):
tortix 8732 0.0 0.0 220008 148 ? S May19 0:00 /var/asl/usr/sbin/tortixd root 11227 0.0 0.0 61252 836 pts/10 S+ 15:24 0:00 grep tortixd root 16369 0.0 0.0 219872 200 ? Ss Apr15 2:21 /var/asl/usr/sbin/tortixd tortix 22527 0.0 0.0 220992 188 ? S May09 0:01 /var/asl/usr/sbin/tortixd tortix 22747 0.0 0.0 221232 188 ? S May09 0:01 /var/asl/usr/sbin/tortixd tortix 22819 0.0 0.0 221492 192 ? S May09 0:01 /var/asl/usr/sbin/tortixd tortix 24961 0.0 0.0 220988 192 ? S May09 0:00 /var/asl/usr/sbin/tortixd
If you do not see the "/var/asl/usr/sbin/tortixd" process running as the tortix user, then the ASL web console server is not running.
If the service is not running, you can start it with this command:
/etc/init.d/tortixd start
If you are missing that file, and you installed ASL you may have told the installer not to install the GUI or someone may have removed this part of ASL from your system. Please re-run the ASL installer and reinstall ASL. The ASL installer is the best tool for installing ASL completely. Instructions for installing ASL are available on this page: ASL_installation.
If the GUI starts correctly you should see this:
Starting tortixd: [ OK ]
If you get this error:
Address already in use: make_sock: could not bind to address 0.0.0.0:30000
Continue to step 5 below.
If you get any other message or error, please check to make sure your system is up to date by using yum to update your system. If you do not keep your system up to date you may need to install many updates that are not related to ASL. We always recommend you keep up with your vendors updates as many of them fix critical security vulnerabilities and to check with your OS vendors and any third party vendors to ensure these updates will work with your OS and third party products. To upgrade your entire system you would run these commands as root:
yum upgrade
asl -s -f
NOTE: If your system is a non-standard hybrid, such as a partial FC 11/12 system or a beta release of a distribution - that configuration is not supported. We can not test hybrids and only support standard releases, so make sure you are running on a supported distribution.
If you still are getting an error starting tortixd, please send us this information:
- The output of the command "/etc/init.d/tortixd start"
- The output of this command "cat /etc/redhat-release"
- Send this file /var/log/httpd/asl_error_log
- And this file /var/log/httpd/asl_ssl_error_log
[edit] Step 5: Make sure you dont have something else listening on port 30000
netstat -anp | grep tortixd | grep 30000
and you should see an output similar to this:
tcp 0 0 :::30000 :::* LISTEN 3547/tortixd
If you do not see the tortixd service listening on port 30000, and see something else listening on that port you will need to disable this other application. Please contact the vendor for that application for assistance with disabling their software.
If you do see tortixd listening on port 30000, and you got this error in step 4:
Address already in use: make_sock: could not bind to address [::]:30000
Or this error:
Address already in use: make_sock: could not bind to address 0.0.0.0:30000
Restart the tortixd service by running these two commands as root:
service tortixd stop
Note: If you do not get tortixd to stop, run this command as root:
killall -9 tortixd
Once tortixd is stopped, restart tortixd by running this command as root:
service tortixd start
If tortixd starts correctly, you will see this output:
Starting tortixd: [ OK ]
If you do not get tortixd to start, continue this troubleshooting process.
[edit] Step 6: Check your firewall rules
If you are using any third party firewall management tools, uninstall them and if you can not uninstall them disable them. You will also want to make sure you do not have any pre-existing firewall rules on your system. This is the most common cause of firewall related issues: pre-existing firewall rules.
[edit] Step 6a: Check your tortixd ACLs
If you have configured ASL to limit access to specific IPs, check this file to ensure your current IP(s) are on the list of custom user defined allowed IPs:
/etc/asl/firewall/tortixd-access-list
the format is one IP or CIDR per line, for example:
1.2.3.4 10.0.0.0/8
If your IP address is not included in this list you will need to add it to the file. Then run this command as root:
service asl-firewall restart
[edit] Step 6b: Check to see if you have any firewall rules
Run this command as the root user (not via sudo):
iptables -L -n
If you see this:
Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination
You have no firewall rules. The problem is not caused by the firewall on your server, continue to step 7.
[edit] Step 6c: Check to see if there are third-party firewall rules that are causing a conflict
By default ASL will generate firewall rules that look like this:
Chain INPUT (policy ACCEPT) target prot opt source destination ASL-GEO-BLACKLIST all -- 0.0.0.0/0 0.0.0.0/0 ASL-BLACKLIST all -- 0.0.0.0/0 0.0.0.0/0 ASL-SMALLPACKETS ah -- 0.0.0.0/0 0.0.0.0/0 length 0:35 ASL-SMALLPACKETS esp -- 0.0.0.0/0 0.0.0.0/0 length 0:49 ASL-SMALLPACKETS 47 -- 0.0.0.0/0 0.0.0.0/0 length 0:39 ASL-SMALLPACKETS 30 -- 0.0.0.0/0 0.0.0.0/0 length 0:31 ASL-SMALLPACKETS icmp -- 0.0.0.0/0 0.0.0.0/0 length 0:27 ASL-SMALLPACKETS tcp -- 0.0.0.0/0 0.0.0.0/0 length 0:39 ASL-SMALLPACKETS udp -- 0.0.0.0/0 0.0.0.0/0 length 0:27 ASL-BADPACKETS tcp -- 0.0.0.0/0 0.0.0.0/0 tcp option=128 ASL-BADPACKETS tcp -- 0.0.0.0/0 0.0.0.0/0 tcp option=64 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x37 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x2B ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x29 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x1A ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x0A ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x0D ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x1C ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x03 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x01 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x29 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x29/0x29 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x22/0x22 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x11/0x01 ASL-FRAGMENTS all -f 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID ASL-TORTIXD-ACL tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:30000 state NEW
All rules in the INPUT chain should start with the word "ASL", as in the example above, with the exception of the INVALID rule:
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
If you see any rules that do not contain the word "ASL" then you have some third party firewall management product that has modified your firewall rules, or policy defaults. You must remove this product from your system as described in the ASL installation pre-requisites page:
https://www.atomicorp.com/wiki/index.php/ASL_prerequisites#firewalls
Please contact the developers of this software for assistance removing it. Once you've removed this product you will want to follow Step 6d below to remove any saved rules from your system and reset your firewall to the ASL defaults.
[edit] Step 6d: Check to make sure you dont have your servers IP addresses blocked in your firewall
Note: If you are using a third party firewall tool, you will need to check with the developers of that application to see if its blocking your servers IP. These instructions will only tell you if ASL is currently blocking your servers IPs.
Run this command script as the root user:
for ip in `ifconfig -a | grep inet | awk -F: '{print $2}' | awk '{print $1}'`; do iptables -L -n |grep $ip | grep ASL-ACTIVE-RESPONSE; done
If you do not see any output then ASL is not blocking your servers IPs.
If you get an output similar to this:
ASL-ACTIVE-RESPONSE all -- 1.2.3.4 0.0.0.0/0
When 1.2.3.4 is one of your servers IP addresses, then you have not whitelisted your servers IP address in ASL.
Please see this FAQ for instructions about how to whitelist an IP, CIDR or set of IP addresses:
https://www.atomicorp.com/wiki/index.php/ASL_FAQ#How_do_you_whitelist_an_IP_in_ASL.3F
[edit] Step 6e: Check to see if there are any ASL generated firewall rules that are causing a conflict
The quickest way to detetermine if your ASL firewall rules are causing an issue is to flush all your firewall rules:
/etc/init.d/asl-firewall stop
If your firewall rules are flushed, you should see this:
Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination
If you see anything other than this, you have some third party firewall management product that has modified your firewall rules, or policy defaults. Please contact the developers of this software for assistance removing it, and restoring your firewall policy defaults.
If you are now able to connect, you had a firewall rule problem. If an ASL generated firewall rule caused this issue it will log this. Check your system logs for any ASL generated firewall rules, and your IP address. Run this command as root, replacing 1.2.3.4 with your IP address.
grep ASL /var/log/messages | grep 1.2.3.4
If you do not see any log events then an ASL generated rule has not caused this issue.
If you do see an ASL event, please check the ASL FAQ for which feature is causing this condition.
If you still can not connect, the problem is not caused by the firewall rules on your server, continue to step 7.
[edit] Step 6f: Reset your firewall rules
If you have been making modifications to your firewall with the ASL firewall manager, you can reset your firewall by following this process:
https://support.atomicorp.com/hc/en-us/articles/360000591068-How-to-reset-the-network-firewall
If you can now connect, then you had configured a firewall rule or rules that were blocking you from accessing the web console. Please see the ASL_firewall for documentation on how to use the ASL firewall manager.
If you still can not connect, the servers firewall is not the cause of your issue, continue to step 7
[edit] Step 7: Run up a sniffer to make sure the problem is not upstream
Note: Make sure you set "eth0" below to the interface that has the IP address assigned to it that you are going to test. If you tell the sniffer to watch a different interface, you will not see the traffic.
You may need to install a sniffer on your system. One sniffer that most Linux distributions include is wireshark another is tcpdump.
[edit] sniffer option 1: wireshark
If you do not have it already installed, this command may help you to install it. You will want to run this command as root:
yum -y install wireshark
On some patforms, the command to run wireshark is "tethereal" on others it is "tshark". Please try both commands. If these do not work for you, please contact your OS vendor for assistance installing a sniffer on your system.
tethereal -i eth0 port 30000
If you do not have tethereal installed, please try these commands:
yum install tethereal
or
yum install wireshark
If you can not install tethereal, see the "tcpdump" example that follows the tethereal example:
tethereal example:
If you have no problems upstream, you will something like this:
Capturing on eth0 0.000000 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=690440808 TSER=0 WS=7 0.000055 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=491123062 TSER=690440808 WS=7 0.047190 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=690440854 TSER=491123062 0.048117 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=114 TSV=690440855 TSER=491123062 0.048149 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [ACK] Seq=1 Ack=115 Win=5888 Len=0 TSV=491123074 TSER=690440855 0.403626 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [ACK] Seq=1 Ack=115 Win=5888 Len=1448 TSV=491123163 TSER=690440855 0.403646 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [PSH, ACK] Seq=1449 Ack=115 Win=5888 Len=159 TSV=491123163 TSER=690440855 0.452383 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [ACK] Seq=115 Ack=1449 Win=8832 Len=0 TSV=690441260 TSER=491123163 0.454911 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [ACK] Seq=115 Ack=1608 Win=11648 Len=0 TSV=690441262 TSER=491123163 0.455637 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [PSH, ACK] Seq=115 Ack=1608 Win=11648 Len=198 TSV=690441263 TSER=491123163 0.455652 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [ACK] Seq=1608 Ack=313 Win=6912 Len=0 TSV=491123176 TSER=690441263 0.457887 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [PSH, ACK] Seq=1608 Ack=313 Win=6912 Len=59 TSV=491123176 TSER=690441263 0.508890 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [PSH, ACK] Seq=313 Ack=1667 Win=11648 Len=565 TSV=690441315 TSER=491123176 0.547198 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [ACK] Seq=1667 Ack=878 Win=8064 Len=0 TSV=491123199 TSER=690441315 1.895139 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [ACK] Seq=1667 Ack=878 Win=8064 Len=1448 TSV=491123535 TSER=690441315 1.895160 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [PSH, ACK] Seq=3115 Ack=878 Win=8064 Len=1042 TSV=491123535 TSER=690441315 1.947302 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [ACK] Seq=878 Ack=4157 Win=17536 Len=0 TSV=690442755 TSER=491123535 1.947693 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [PSH, ACK] Seq=878 Ack=4157 Win=17536 Len=37 TSV=690442755 TSER=491123535 1.947715 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [ACK] Seq=4157 Ack=915 Win=8064 Len=0 TSV=491123549 TSER=690442755 1.948027 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [FIN, ACK] Seq=915 Ack=4157 Win=17536 Len=0 TSV=690442755 TSER=491123535 1.949539 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [PSH, ACK] Seq=4157 Ack=916 Win=8064 Len=37 TSV=491123549 TSER=690442755 1.949610 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [FIN, ACK] Seq=4194 Ack=916 Win=8064 Len=0 TSV=491123549 TSER=690442755 1.997338 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [RST] Seq=916 Win=0 Len=0
[edit] sniffer option 2: tcpdump
If you do not have wireshark available for your platform, most OS vendors will include tcpdump by default. If tcpdump is not installed on your system, please run this command as root to install it:
yum -y install tcpdump
If this does not work for your OS, please contact your OS vendor for assistance installing a sniffer.
tcpdump example:
If you have no problems upstream, you will see something like this:
1.2.3.4 is your client 9.8.7.6 is your server
10:34:26.227472 IP 1.2.3.4.54405 > 9.8.7.6.30000: S 3615766533:3615766533(0) win 5840 <mss 1460,sackOK,timestamp 771590282 0,nop,wscale 7> 10:34:26.227514 IP 9.8.7.6.30000 > 1.2.3.4.54405: S 1843855213:1843855213(0) ack 3615766534 win 5792 <mss 1460,sackOK,timestamp 511410428 771590282,nop,wscale 7> 10:34:26.274981 IP 1.2.3.4.54405 > 9.8.7.6.30000: . ack 1 win 46 <nop,nop,timestamp 771590329 511410428> 10:34:26.275774 IP 1.2.3.4.54405 > 9.8.7.6.30000: P 1:147(146) ack 1 win 46 <nop,nop,timestamp 771590330 511410428> 10:34:26.275806 IP 9.8.7.6.30000 > 1.2.3.4.54405: . ack 147 win 54 <nop,nop,timestamp 511410440 771590330> 10:34:26.276150 IP 9.8.7.6.30000 > 1.2.3.4.54405: P 1:139(138) ack 147 win 54 <nop,nop,timestamp 511410440 771590330> 10:34:26.327459 IP 1.2.3.4.54405 > 9.8.7.6.30000: . ack 139 win 54 <nop,nop,timestamp 771590382 511410440> 10:34:26.329933 IP 1.2.3.4.54405 > 9.8.7.6.30000: P 147:723(576) ack 139 win 54 <nop,nop,timestamp 771590383 511410440> 10:34:26.369824 IP 9.8.7.6.30000 > 1.2.3.4.54405: . ack 723 win 63 <nop,nop,timestamp 511410464 771590383> 10:34:26.518491 IP 9.8.7.6.30000 > 1.2.3.4.54405: . 139:1587(1448) ack 723 win 63 <nop,nop,timestamp 511410501 771590383> 10:34:26.518514 IP 9.8.7.6.30000 > 1.2.3.4.54405: P 1587:2629(1042) ack 723 win 63 <nop,nop,timestamp 511410501 771590383> 10:34:26.518619 IP 9.8.7.6.30000 > 1.2.3.4.54405: P 2629:2666(37) ack 723 win 63 <nop,nop,timestamp 511410501 771590383> 10:34:26.518660 IP 9.8.7.6.30000 > 1.2.3.4.54405: F 2666:2666(0) ack 723 win 63 <nop,nop,timestamp 511410501 771590383> 10:34:26.570119 IP 1.2.3.4.54405 > 9.8.7.6.30000: . ack 2629 win 100 <nop,nop,timestamp 771590625 511410501> 10:34:26.570586 IP 1.2.3.4.54405 > 9.8.7.6.30000: P 723:760(37) ack 2667 win 100 <nop,nop,timestamp 771590625 511410501> 10:34:26.570654 IP 9.8.7.6.30000 > 1.2.3.4.54405: . ack 760 win 63 <nop,nop,timestamp 511410514 771590625> 10:34:26.570760 IP 1.2.3.4.54405 > 9.8.7.6.30000: R 760:760(0) ack 2667 win 100 <nop,nop,timestamp 771590625 511410501> 10:34:26.617628 IP 1.2.3.4.54406 > 9.8.7.6.30000: S 3610239263:3610239263(0) win 5840 <mss 1460,sackOK,timestamp 771590673 0,nop,wscale 7> 10:34:26.617660 IP 9.8.7.6.30000 > 1.2.3.4.54406: S 1858764235:1858764235(0) ack 3610239264 win 5792 <mss 1460,sackOK,timestamp 511410525 771590673,nop,wscale 7> 10:34:26.660089 IP 1.2.3.4.54407 > 9.8.7.6.30000: S 3617695320:3617695320(0) win 5840 <mss 1460,sackOK,timestamp 771590715 0,nop,wscale 7> 10:34:26.660111 IP 9.8.7.6.30000 > 1.2.3.4.54407: S 1850303361:1850303361(0) ack 3617695321 win 5792 <mss 1460,sackOK,timestamp 511410536 771590715,nop,wscale 7> 10:34:26.664975 IP 1.2.3.4.54406 > 9.8.7.6.30000: . ack 1 win 46 <nop,nop,timestamp 771590719 511410525> (and a lot more)
If you dont see a packet exchange, the problem is not with your server - its upstream at some other firewall, or even on your desktop or with your home or office firewall.
If you do see all of this, and still can't connect - you ARE connecting - check your browser to make sure its not breaking on the connection. For example, if you connect with "http" instead of "https", or you are connecting to the wrong port, port 3000 instead of 30000.
[edit] Step 8: Check to make sure a third party product has not changed ASL's permissions
Check that the GUI's php's session path:
/var/asl/session
is owned by 'tortix:apache' with permissions 770.
The permissions and ownership should be:
drwxrwx--- 2 tortix apache
[edit] Step 9: Check to make sure tortix is in the apache group
Run this command as root:
grep ^apache: /etc/group | grep tortix
If tortix is not in the group, you will get a blank response. For example:
[root@host]# grep ^apache: /etc/group | grep tortix
[root@host]#
If tortix is in the group, you will see a response similar to this one:
[root@host]# grep ^apache: /etc/group | grep tortix
apache:x:48:apache,tortix
Note: The use of add-ons to apache that change the ownership of temporary files and log files, such as mod_ruid2 may break the logging system. These add-ons are not supported with ASL.
If none of this resolves your issue, please contact support. We would be happy to help you resolve this.
[edit] Not getting any emails from ASL
If you not getting any emails from ASL this can be cause by any number of factors, your mail server is down, your have spam filtering software thats blocking ASL, you have firewall rules that are blocking the connection, etc. To assist with the process of troubleshooting these we have put together this check list. Please run through each step.
Note: Run all of the commands in the steps below as the root user. Do not use sudo.
[edit] Step 1: Ensure that you have your system configured for ASL
Please make sure your system meets and is configured correctly for all the pre-requistes for ASL. These are documented on the ASL_prerequisites page.
[edit] Step 2: Check to make sure the ASL is installed
Run this command as root:
asl -v
If the ASL is installed, you will see output similar to this:
ASL Version 3.2.15-33.el5.art: CentOS 5 (SUPPORTED)
If the ASL is not installed, you will see output similar to this:
bash: asl: command not found...
If ASL is not installed, please see the ASL_installation page for instructions to install ASL.
[edit] Step 3: Check to make sure all ASL updates are installed
First, check to make sure a third party product has not excluded any ASL packages from installation or upgrade. You will want to check your /etc/yum.conf file to make sure it is not globally excluding things incorrectly. For example, some third party products that do not use software and package management will try to exclude the systems PHP packages from install/upgrade by incorrectly using a regular expression such as "php*" or "*php*". Disable all excludes.
Run these commands as root:
aum -uf
asl -s -f
[edit] Step 4: Check to make sure OSSEC is working correctly
Run this command as root:
ps auxwww | grep ossec
You should see an output similar to this:
root 3192 0.0 0.0 50352 112 ? S Jun25 0:01 /var/ossec/bin/ossec-execd ossecm 19622 0.0 0.0 91360 668 ? S Aug04 0:10 /var/ossec/bin/ossec-dbd ossecm 19627 0.0 0.0 24520 352 ? S Aug04 0:04 /var/ossec/bin/ossec-maild ossec 19637 1.5 0.6 47176 6424 ? S Aug04 215:45 /var/ossec/bin/ossec-analysisd root 19641 0.0 0.0 38600 364 ? S Aug04 0:44 /var/ossec/bin/ossec-logcollector root 19652 0.1 1.7 99032 17852 ? S Aug04 26:48 /var/ossec/bin/ossec-syscheckd ossec 19654 0.0 0.0 61520 364 ? S Aug04 0:00 /var/ossec/bin/ossec-monitord
If everything is working correctly, you should see all of these processes running:
- ossec-execd
- ossec-dbd
- ossec-maild
- ossec-analysisd
- ossec-logcollector
- ossec-syscheckd
- ossec-monitord
If any of these processes are not running, please try restarting ossec-hids by running this command as root:
service ossec-hids restart
If OSSEC is not starting, or some of these services are not starting, please see this article for additional system checks to perform to find the root cause of this condition and solutions to apply to address them:
[edit] Step 5: Check your email settings in ASL
[edit] Step 5a
Ensure that you have these settings set to your email address, and that this email address is valid and that your server can resolve the domain:
https://www.atomicorp.com/wiki/index.php/ASL_Configuration#EMAIL
https://www.atomicorp.com/wiki/index.php/ASL_Configuration#PSMON_EMAIL
https://www.atomicorp.com/wiki/index.php/ASL_Configuration#OSSEC_EMAIL
[edit] Step 5b: check to ensure the domain resolves to a working mail server
Check your server to ensure it can resolve the domain name, and that what its resolving to is the correct server. For example, if your email address was somaddress@atomicorp.com, on your server you would check to ensure your server can resolve that domain with this command:
host -t mx yourdomain.com
Replace "yourdomain.com" with the domainname you used in those settings. For example, if you set EMAIL to "user@example.com, your domainname is "example.com" and you would test that your server can resolve that domain name with this command:
host -t mx example.com
If your server can resolve the domain name you will see output similar to this:
host -t mx example.com
example.com mail is handled by 10 mail.example.com
If your server can not resolve the domain name you will see output similar to this:
host -t mx example.com
example.com has no MX record
[edit] Step 5c: Check to ensure your server can resolve the mail servers IP address
Then check to make sure your server can resolve the mail servers IP address. For example, in the above the mail servers name is "mail.example.com". You will need to ensure that this resolves on your server, which this command:
nslookup mail.example.com
Server: 127.0.0.1 Address: 127.0.0.1#53 Non-authoritative answer: Name: mail.example.com Address: 1.2.3.4
And then you would need to ensure the IP address returned is in fact the correct IP address to send email to. If that is not correct you will need to fix your DNS server(s) to return the correct IP address. Please contact your DNS vendor for assistance with configuring your DNS server. If your server can not resolve domain name, contact your DNS vendor for assistance.
[edit] Step 6: Ensure that you have OSSEC configured to send emails
This setting:
https://www.atomicorp.com/wiki/index.php/ASL_Configuration#OSSEC_NOTIFY
Must be set to yes.
And make sure this setting:
https://www.atomicorp.com/wiki/index.php/ASL_Configuration#OSSEC_MAX_MSG
Is set to a positive integer between 1 and 9999. This will allow ASL to send emails as they occur and at the amount per hour you have set in this field. If you put a number outside of this range in this field, it will cause the maild service to shut off and you will not receive email alerts. We recommend you keep this set to "1". Note: By default, only alerts that are a level 7 or higher are emailed.
[edit] Step 7: Ensure that OSSEC is enabled
This must be set to yes:
https://www.atomicorp.com/wiki/index.php/ASL_Configuration#OSSEC_ENABLED
[edit] Step 8: Ensure that OSSEC is set to server mode
OSSEC will only send emails if it is set to "server" mode:
https://www.atomicorp.com/wiki/index.php/ASL_Configuration#OSSEC_MODE
[edit] Step 9: Ensure the SMTP server is correct
[edit] Step 9a
This setting:
https://www.atomicorp.com/wiki/index.php/ASL_Configuration#OSSEC_SMTP_SERVER
Tells OSSEC what server to use to send email. This server must allow OSSEC to send email. Normally this would be the server itself, and you would use the IP address 127.0.0.1. You must be running a mail server on your server to use 127.0.0.1, and it must be configured correctly. Please contact your mail server vendor for assistance with configuring a mail server.
Note: If you use a hostname, and not an IP address, you must ensure that your server can correctly resolve this IP address. The OS will do the resolution, so if your system is unable to determine the IP address for your OSSEC_SMTP_SERVER, contact your OS and DNS vendors for assistance. That means something is wrong with their your DNS or OS.
You can test to see if you can connect to your mail server with the command "telnet", in the example below we will connect to the 127.0.0.1 mailserver. Only port 25 is supported with OSSEC.
telnet 127.0.0.1 25
telnet 127.0.0.1 25 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. 220 servername.example.com ESMTP Postfix
If your mail sever is listening on that IP address you will see something similar to the example above.
If you do not see something similar to this, for example you see nothing, then your mail server is not running on that IP address, or you can not connect to that IP on that port, move onto step 10.
[edit] Step 9b
Check to make sure you have a mail server running on that system. If you are using 127.0.0.1 as your mail server, you can run this command as root to see if you have a mail server running on your local system:
netstat -an | grep LISTEN | grep :25
If you have a local mail server running you will see output similar to this:
If you do not, then you do not have a local mail server running. Please contact your mail server vendor for assistance setting up a mail server, or use a remote mail server that will accept email from your server.
If you do have a mail server running locally, and you can not connect to it, please proceed to Step 10 to check your firewall rules.
If you have a remote mail server, please proceed to Step 10.
[edit] Step 10: Check your firewall rules
If you are using any third party firewall management tools, uninstall them and if you can not uninstall them disable them. You will also want to make sure you do not have any pre-existing firewall rules on your system. This is the most common cause of firewall related issues: pre-existing firewall rules.
[edit] Step 10a: Check to see if you have any firewall rules
Run this command as the root user (not via sudo):
iptables -L -n
If you see this:
Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination
You have no firewall rules. The problem is not caused by the firewall on your server, continue to step 11.
[edit] Step 10b: Check to see if there are third-party firewall rules that are causing a conflict
By default ASL will generate firewall rules that look like this:
Chain INPUT (policy ACCEPT) target prot opt source destination ASL-GEO-BLACKLIST all -- 0.0.0.0/0 0.0.0.0/0 ASL-BLACKLIST all -- 0.0.0.0/0 0.0.0.0/0 ASL-SMALLPACKETS ah -- 0.0.0.0/0 0.0.0.0/0 length 0:35 ASL-SMALLPACKETS esp -- 0.0.0.0/0 0.0.0.0/0 length 0:49 ASL-SMALLPACKETS 47 -- 0.0.0.0/0 0.0.0.0/0 length 0:39 ASL-SMALLPACKETS 30 -- 0.0.0.0/0 0.0.0.0/0 length 0:31 ASL-SMALLPACKETS icmp -- 0.0.0.0/0 0.0.0.0/0 length 0:27 ASL-SMALLPACKETS tcp -- 0.0.0.0/0 0.0.0.0/0 length 0:39 ASL-SMALLPACKETS udp -- 0.0.0.0/0 0.0.0.0/0 length 0:27 ASL-BADPACKETS tcp -- 0.0.0.0/0 0.0.0.0/0 tcp option=128 ASL-BADPACKETS tcp -- 0.0.0.0/0 0.0.0.0/0 tcp option=64 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x37 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x2B ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x29 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x1A ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x0A ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x0D ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x1C ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x03 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x01 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x29 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x29/0x29 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x22/0x22 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x11/0x01 ASL-FRAGMENTS all -f 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID ASL-TORTIXD-ACL tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:30000 state NEW
All rules in the INPUT chain should start with the word "ASL", as in the example above, with the exception of the INVALID rule:
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
If you see any rules that do not contain the word "ASL" then you have some third party firewall management product that has modified your firewall rules, or policy defaults. You must remove this product from your system as described in the ASL installation pre-requisites page:
https://www.atomicorp.com/wiki/index.php/ASL_prerequisites#firewalls
Please contact the developers of this software for assistance removing it. Once you've removed this product you will want to follow Step 6d below to remove any saved rules from your system and reset your firewall to the ASL defaults.
[edit] Step 10c: Check to make sure you dont have your servers IP addresses blocked in your firewall
Note: If you are using a third party firewall tool, you will need to check with the developers of that application to see if its blocking your servers IP. These instructions will only tell you if ASL is currently blocking your servers IPs.
Run this command script as the root user:
for ip in `ifconfig -a | grep inet | awk -F: '{print $2}' | awk '{print $1}'`; do iptables -L -n |grep $ip | grep ASL-ACTIVE-RESPONSE; done
If you do not see any output then ASL is not blocking your servers IPs.
If you get an output similar to this:
ASL-ACTIVE-RESPONSE all -- 1.2.3.4 0.0.0.0/0
When 1.2.3.4 is one of your servers IP addresses, then you have not whitelisted your servers IP address in ASL.
Please see this FAQ for instructions about how to whitelist an IP, CIDR or set of IP addresses:
https://www.atomicorp.com/wiki/index.php/ASL_FAQ#How_do_you_whitelist_an_IP_in_ASL.3F
[edit] Step 10d: Check to see if there are any ASL generated firewall rules that are causing a conflict
The quickest way to detetermine if your ASL firewall rules are causing an issue is to flush all your firewall rules:
service asl-firewall stop
If your firewall rules are flushed, you should see this when you run the command "iptables -L -n".
Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination
If you see anything other than this, you have some third party firewall management product that has modified your firewall rules, or policy defaults. Please contact the developers of this software for assistance removing it, and restoring your firewall policy defaults.
If you are now able to connect, (step 9), you had a firewall rule problem. If an ASL generated firewall rule caused this issue it will log this. Check your system logs for any ASL generated firewall rules, and your IP address. Run this command as root, replacing 1.2.3.4 with your IP address.
grep ASL /var/log/messages | grep 1.2.3.4
If you do not see any log events then an ASL generated rule has not caused this issue.
If you do see an ASL event, please check the ASL FAQ for which feature is causing this condition.
If you still can not connect, the problem is not caused by the firewall rules on your server, continue to step 11.
[edit] Step 10e: Reset your firewall rules
If you have been making modifications to your firewall with the ASL firewall manager, you can reset your firewall rules to the defaults by running these commands as the root user (not via sudo):
cp /etc/asl/firewall/running.fw /root
rm /etc/asl/firewall/running.fw
asl -s -f
If you can now connect, then you had configured a firewall rule or rules that were blocking you from accessing the web console. Please see the ASL_firewall for documentation on how to use the ASL firewall manager.
If you still can not connect, the servers firewall is not the cause of your issue, continue to step 7.
NOTE: You can restore your custom firewall rules, only if you used the commands above to clear your firewall rules by running these commands as the root user:
cp /root/running.fw /etc/asl/firewall/running.fw
asl -s -f
[edit] Step 10f: Send a test email
If you have confirmed your firewall rules are not blocking you from sending email to your mail server, send a test email using the method below:
1) Connect to the mail server, change 127.0.0.1 to the IP address of your server
telnet 127.0.0.1 25
2) The server will respond with something similar to this:
Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. 220 server ESMTP Postfix
If you do not get a response, then there is a problem with your network, please proceed to step 11.
If you do get a response, please continue with 3 below.
3) Send the "helo" response with your servers name, for example if your servers name is www.example.com send this:
helo www.example.com
4) The server will respond with this, if it does not, contact your mail server vendor for assistance your mail server is not setup correctly:
250 yourmailserversnames
5) Send the mail from line using the setting you configured in the OSSEC_FROM setting.
mail from: someuser@atomicorp.com
6) If your mail server is setup correctly it will respond with the below, if it doesnt not contact your mail server vendor for assistance your mail server is not setup correctly:
250 2.1.0 Ok
7) Send the rcpt to line using the setting you configured in the OSSEC_EMAIL setting.
rcpt to: someuser@example.com
8) If your mail server is setup correctly it will respond with the below, if it doesnt not contact your mail server vendor for assistance your mail server is not setup correctly:
250 2.1.5 Ok
9) Then send a quick test message, first type "data" and hit enter:
data
10) If your mail server is setup correctly it will respond with a similiar response to the below, if it doesnt not contact your nail server vendor for assistance your mail server is not setup correctly:
354 End data with <CR><LF>.<CR><LF>
11) Type in a quick test message
test message
12) And then on a new line type . and hit enter as below:
.
13) If your mail server is setup correctly it will respond with a similiar response to the below, if it doesnt not contact your nail server vendor for assistance your mail server is not setup correctly:
250 2.0.0 Ok: queued as D5D6A10D8112
Then type "quit" to log out of your mail server.
If you got a similar message as above, ASL is able to send mail to your mail server. If you do not get this email, check your mail servers logs and your antispam settings, as this means your mail server is not sending these emails to your email address. Contact your nail server vendor for assistance your mail server is not setup correctly.
[edit] Step 11: Run up a sniffer to make sure the problem is not upstream
Note: Make sure you set "eth0" below to the interface that has the IP address assigned to it that you are going to test. If you tell the sniffer to watch a different interface, you will not see the traffic.
You may need to install a sniffer on your system. One sniffer that most Linux distributions include is wireshark another is tcpdump.
[edit] sniffer option 1: wireshark
If you do not have it already installed, this command may help you to install it. You will want to run this command as root:
yum -y install wireshark
On some patforms, the command to run wireshark is "tethereal" on others it is "tshark". Please try both commands. If these do not work for you, please contact your OS vendor for assistance installing a sniffer on your system.
Note: change 1.2.3.4 to the IP address for your mailserver. If your mailserver is 127.0.0.1, then change -i eth0 to -i lo.
tethereal -i eth0 port 25 and hostname 1.2.3.4
If you do not have tethereal installed, please try these commands:
yum install tethereal
or
yum install wireshark
If you can not install tethereal, see the "tcpdump" example that follows the tethereal example:
tethereal example:
If you have no problems upstream, you will something like this:
0.000000 127.0.0.1 -> 127.0.0.1 TCP 50849 > smtp [SYN] Seq=0 Win=32792 Len=0 MSS=16396 TSV=596718574 TSER=0 WS=8 0.000014 127.0.0.1 -> 127.0.0.1 TCP smtp > 50849 [SYN, ACK] Seq=0 Ack=1 Win=32768 Len=0 MSS=16396 TSV=596718574 TSER=596718574 WS=8 0.000025 127.0.0.1 -> 127.0.0.1 TCP 50849 > smtp [ACK] Seq=1 Ack=1 Win=33024 Len=0 TSV=596718574 TSER=596718574 0.176700 127.0.0.1 -> 127.0.0.1 SMTP S: 220 yourserversname ESMTP Postfix 0.176713 127.0.0.1 -> 127.0.0.1 TCP 50849 > smtp [ACK] Seq=1 Ack=40 Win=33024 Len=0 TSV=596718751 TSER=596718751
[edit] sniffer option 2: tcpdump
If you do not have wireshark available for your platform, most OS vendors will include tcpdump by default. If tcpdump is not installed on your system, please run this command as root to install it:
yum -y install tcpdump
If this does not work for your OS, please contact your OS vendor for assistance installing a sniffer.
tcpdump example:
If you have no problems upstream, you will see something like this:
1.2.3.4 is your client 9.8.7.6 is your server
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lo, link-type EN10MB (Ethernet), capture size 96 bytes 13:06:11.808773 IP 1.2.3.4.40930 > 9.8.7.6.smtp: S 3226807163:3226807163(0) win 32792 <mss 16396,sackOK,timestamp 597368964 0,nop,wscale 8> 13:06:11.808790 IP 9.8.7.6..smtp > 1.2.3.4.40930: S 3784740604:3784740604(0) ack 3226807164 win 32768 <mss 16396,sackOK,timestamp 597368964 597368964,nop,wscale 8> 13:06:11.808801 IP 1.2.3.4.40930 > 9.8.7.6..smtp: . ack 1 win 129 <nop,nop,timestamp 597368964 597368964> 13:06:11.950623 IP 9.8.7.6..smtp > 1.2.3.4.40930: P 1:40(39) ack 1 win 128 <nop,nop,timestamp 597369106 597368964> 13:06:11.950636 IP 1.2.3.4.40930 > 9.8.7.6..smtp: . ack 40 win 129 <nop,nop,timestamp 597369106 597369106> (and a lot more)
[edit] Step 12
If you dont see a packet exchange, the problem is with the mail server itself. Either your mail server does not accept connections, is not running a mail server, you have an upstream firewall that is blocking you (if you are using a remote mailserver) or you have a network problem that is preventing you from connecting to your mail server. Please contact your mail server vendor for assistance with connections issues with your mailserver.
If everything is working, and you have confirmed that your mail server is listening, and you can successfully send a test message, then check your mail servers logs. This means that ASL is able to send email, and your mail server is just not sending it on your account. Please contact your mail server vendor for assistance with configuring your mail server to send email to your email account.
If you require additional assistance with this issue, please send us the output of all these steps as we will need this information to assist you. Please do not contact us about issues concerning the correct configuration of your mail server. Please contact your mail server vendor for assistance with their products.
[edit] Step 12a
This step is provided only as a courtesy. If your local mail server is not working, contact your mail server vendor for assistance. These instructions are provided without any support.
In some cases, your local mailserver (if you have configured ASL to use a local mailserver) may not be running on working. Restarting your local mailserver may resolve this issue.
[edit] postfix
To restart postfix run this command:
service postfix restart
[edit] qmail
To restart qmail, run this command:
service qmail restart
[edit] ASL web gui not running
[edit] Step 1. Check to make sure its installed
rpm -qa | grep tortixd
If you do not see that package installed your system may be missing other components of ASL as well. You should run "yum upgrade" to make sure your system is up to date.
Then run the official ASL installer:
wget -q -O - https://www.atomicorp.com/installers/asl |sh
[edit] Step 2: Start the service
/etc/init.d/tortixd start
[edit] Step 3: Follow the steps above in "Can not connect to port 3000" to make sure your firewall and upstream (if any) is configured properly
You can also re-install the web GUI with this command:
yum reinstall asl-web
[edit] Empty Gui
This means that a third party application or a user has misconfigured your ASL directories to prevent tortixd from opening the files it needs to render the GUI.
Check to make sure the permissions on this directory:
/var/asl/session
Have not been changed. The permissions should be:
drwxrwx--- 2 root apache 4096 Jul 29 14:55 session
In general, if a third party application or user changes these permissions ASL may be able to fix this by running this command:
asl -s -f
The tortix user should also be in the apache group.
usermod -a -G apache tortix
[edit] No events in the ASL GUI
[edit] Step 1: Test ASL Web
First check to make sure that ASL Web is working. In the Recent Events tab of the Security Events window, set the level filter to 1 so you can see all events. By default ASL Web is set to a higher level to filter out lower importance events and to only display attacks and highly supicious events by default. If the ASL Web is showing lower level events, you may either not have any attacks or suspicious events to report, or you may have disabled or not installed all of ASL protection components.
In that case, send a simple test from the local system, or another system that should be both blocked and logged. This article provides a test example:
Atomic_ModSecurity_Rules#Testing_to_see_if_the_rules_are_loaded
[edit] Step 2: Check to make sure ASL is up to date
If you are running an older version of ASL, please see this article to upgrade.
If you are running the current major revision of ASL, run these commands as root:
- /var/asl/bin/aum -u
- /var/asl/bin/asl -s -f
[edit] Step 3: Verify that OSSEC is working correctly
Run this command as root:
ps auxwww | grep ossec
You should see an output similar to this:
root 3192 0.0 0.0 50352 112 ? S Jun25 0:01 /var/ossec/bin/ossec-execd ossecm 19622 0.0 0.0 91360 668 ? S Aug04 0:10 /var/ossec/bin/ossec-dbd ossecm 19627 0.0 0.0 24520 352 ? S Aug04 0:04 /var/ossec/bin/ossec-maild ossec 19637 1.5 0.6 47176 6424 ? S Aug04 215:45 /var/ossec/bin/ossec-analysisd root 19641 0.0 0.0 38600 364 ? S Aug04 0:44 /var/ossec/bin/ossec-logcollector root 19652 0.1 1.7 99032 17852 ? S Aug04 26:48 /var/ossec/bin/ossec-syscheckd ossec 19654 0.0 0.0 61520 364 ? S Aug04 0:00 /var/ossec/bin/ossec-monitord
If everything is working correctly, you should see all of the above processes running.
If any of these processes are not running, please try restarting ossec-hids by running this command as root:
- service ossec-hids restart
If OSSEC is not starting, or some of these services are not starting, please see this article for additional system checks to perform to find the root cause of this condition and solutions to apply to address them:
[edit] Step 4: Verify that mysql is working correctly
[edit] Check for errors in the mysql log
Some products, such as cpanel, configure mysql to not save logs. Please contact your database vendor if this is your case for advice on how to configure your database server to log errors as you will need to confirm the database is working correctly before moving on to the next step.
Mysql is normally configured to log errors to /var/log/mysqld.log. A corrupt database will result in an error in the mysql logs. A typical error may be one of the following (please check your logs for any mysql errors and not just the example below):
- Table './tortix/alert' is marked as crashed and should be repaired
If you see errors such as these, please see this faq article for links to mysql procedures for fixing databases.
- [Warning] Aborted connection #### to db: 'tortix' user: 'tortix' ... (Got an error reading communication packets)
This message typically indicates that the connection was closed after timing out. If you see errors like these, it may indicate that you have mysql configured with a low wait_timeout. Restoring this setting to its default value of 28800 is recommended.
[edit] Make sure mysql is listening on the IP and port you configured
A database that is not listening on a TCP port or the same port that you have configured ASL to use. Check to make sure your MySQL server is listening on port 3306 on the IP address you have configured ASL to use as its database server.
Additionally, a database server that is listening on a different IP address from the one configured for ASL can cause start up errors. Check to make sure your MySQL server is listening on port 3306 on the IP address you have configured ASL to use as its database server.
Please see this setting:
https://www.atomicorp.com/wiki/index.php/ASL_Configuration#OSSEC_DATABASE_SERVER
And ensure you can connect to port 3306 on that configured IP or hostname.
[edit] Check to make sure the tortix mysql user permissions are correct
If you are receiving an error during database setup similar to this:
Loading ASL Web database schema: ERROR 1227 (42000) at line 27: Access denied; you need the SUPER privilege for this operation
then you need to check the MySQL permissions. Specifically if the tortix mysql user is either unable to create or is unable to execute triggers, events will not show up properly.
To resolve this, follow these steps:
- log in to mysql as the tortix user
- mysql -u `cat /etc/asl/config | grep OSSEC_DATABASE_USERNAME | awk -F\" '{print $2}'` -p`cat /etc/asl/config | grep OSSEC_DATABASE_PASSWORD | awk -F\" '{print $2}'`
- Once you are logged into mysql successfully, select the ASL database with this command:
use tortix;
Note: If you changed the default name for the ASL database, you will need to change this accordingly.
- verify that triggers have been created
- Type this command in the mysql prompt:
SHOW TRIGGERS;
If no triggers are displayed, the tortix user does not have correct permissions. The following steps should correct the problem:
- Login to mysql as the root / admin user
- Type the following commands in the mysql prompt:
GRANT ALL ON tortix.* TO 'tortix'@'localhost' IDENTIFED BY '[tortix user's password]';
GRANT SUPER ON *.* TO 'tortix'@'localhost' IDENTIFED BY '[tortix user's password]';
Please note the following:- The tortix user's password may be found in /etc/asl/config under the variable OSSEC_DATABASE_PASSWORD
- Check to see what IP mysql is bound to. If it is bound to an ip address other than 127.0.0.1, or mysql is skipping name resolution, then you will need to set the priviliges for the tortix user to use the IP address that mysql is bound to. Replace 'localhost' with the IP address in the above GRANT statements if this is the case.
- Type this command in the mysql prompt:
[edit] Run ASL Web validation
After any of the corrective actions above, or if none of the above resolved the problem, run the following command:
/var/asl/bin/asl --web_validate
Data will be available in ASL Web on the first polling after correction, assuming any new events have come in, it does not require a refresh or relogging. Event data that existed prior to the correction will not be repaired.
[edit] Contact Support
If none of the above resolved the problem, please contact support and install our support keys as shown at the below link and we will have a look.
[edit] Firewall
[edit] FTP not working
[edit] Background
FTP is inarguably the most complex protocol in use on the Internet today, if not of all time, it is also the oldest. Unlike other protocols, like SMTP and HTTP, FTP does not work over one port or even a predicable set of ports. FTP uses lots and lots of ports, and it dynamically selects the ports it uses both of the client and server side randomly. You might think FTP just works over port 21, but thats only the tip of the iceberg. FTPs actual "data" connections happen on random dynamically allocated ports that are unique for each client and connection! Which makes it rather difficult for any firewall to support it. To support FTP all firewalls require what are sometimes referred to as "helper" modules. These modules actually look for what appears to be FTP traffic on all those random, dynamic ports. And they have to do this passively, and in real time because the FTP server has no way to tell the firewall what ports it needs open. The firewall has to figure this out as it goes: dynamically opening and closing the ports that FTP needs to use when you are using any firewall. Without these modules FTP simply doesnt work. And, because FTP is unlike any other protocol in use on your system, thats why its important that not only your firewall rules be setup correctly for FTP, but that both your firewall and your operating system support it, understand it and manage the connections in real time correctly. The article below provides a brief overview of how FTP works:
https://www.centos.org/docs/5/html/Virtual_Server_Administration/s2-ftp-proto-VSA.html
[edit] Solutions
[edit] If you are using the ASL kernel
Step 0) Make sure you are not using a third party firewall
This includes the "iptables" service and firewalld. Make sure you have removed and disabled any third party firewalls from your system. This guide is only for the use of the ASL firewall. If you are using a third party firewall, please contact the vendor for that firewall for assistance.
https://www.atomicorp.com/wiki/index.php/ASL_prerequisites#firewalls
Step 1) Make sure you have port 21 open.
If you are using Fast mode, see this setting:
https://www.atomicorp.com/wiki/index.php/ASL_firewall#FW_INBOUND_TCP_SERVICES
Step 2) If you are specifically using passive mode
You will need to allow in the range of ports you configured your FTP server to use for passive mode. ASL does not configure any FTP server to use passive mode. This can only happen in someone has manually configured your FTP server to use passive mode. If you are not sure what this means, you probably are not using passive mode.
See this setting:
https://www.atomicorp.com/wiki/index.php/ASL_firewall#FW_INBOUND_TCP_SERVICES
And set the range of ports you have configured for passive mode. For example, if you have configured proftp to use this range:
PassivePorts 40110 40310
Then you need to add this to your list of open ports you configured in ASL:
40110:40310
Note: If you have setup your own custom rules, and are not using Fast mode, then you will need to review the Firewall documentation at the URL below:
https://www.atomicorp.com/wiki/index.php/ASL_firewall
If you still can not access FTP, check to ensure you are not using any third party firewalls, including upstream firewalls which would also be blocking this port, or do not support FTP. If you are using a third party firewall, please contact the vendor for that firewall for assistance.
[edit] If you are not using the ASL kernel
Step 0) Make sure you are not using a third party firewall
This includes the "iptables" service. Make sure you have removed and disabled any third party firewalls from your system. This guide is only for the use of the ASL firewall. If you are using a third party firewall, please contact the vendor for that firewall for assistance.
https://www.atomicorp.com/wiki/index.php/ASL_prerequisites#firewalls
Step 1) check to make sure kernel FTP modules loaded
Check to make sure your third party kernel includes the standard Linux FTP tracking modules. All Linux firewalls require special kernel modules to track FTPs data connections (FTP is a very complex protocol), without these modules no Linux firewall will work. These modules have been included with Linux kernels for over a decade, so your kernel should have these kernel modules.
You can check to see if they are loaded with this command run as root:
lsmod | grep ftp
You should see these modules loaded:
nf_nat_ftp
nf_conntrack_ftp
If they are not, you can load them with the standard Linux kernel module command:
modprobe nf_nat_ftp modprobe nf_conntrack_ftp
If you have a really old kernel you may not have these kernel modules, in which case you will need to load the older kernel modules
modprobe ip_conntrack_ftp
modprobe ip_nat_ftp
If your kernel is locked (that is it wont allow modules to be installed once the boot sequence is finished), please see this article:
If you are unsure about how to load kernel modules in Linux, on boot, please see this article:
Installing_custom_kernel_modules_with_ASL
If these commands do not work, please ask your kernel vendor to include the standard Linux FTP modules and for instructions for how to load them. All Linux firewalls require these modules to do support FTP with a firewall, or use the ASL kernel which will always load these standard Linux kernel firewall modules.
[edit] Step 3) If you are using encrypted FTP
As explained above, FTP is a very complex protocol that works over multiple dynamic ports. When Linux is configured to use its kernel based firewall, it uses special "helper" modules to detect the dynamic FTP data connections. These connections do not occur over port 21 or 20, they are dynamic "high ports" (e.g. ports 1024 and higher). The client and server negotiate these (either the client sends the ports it wishes to use, or the server sends the port it wishes to use). When this occurs over an encrypted FTP session Linux can not see this negotiation happen (because its encrypted), and therefore will not dynamically open and close these high port data connections between the server and the client. Therefore, you can not use encrypted FTP with any firewall without permanently opening a range of ports on your firewall(s) for your FTP server(s) to use, and configuring your FTP server(s) to only use that range of ports (this is called "passive mode"), instead of dynamic picking ports (also called "active mode"). Please contact your FTP vendor for assistance with configuring your FTP server. Once you have correctly configured your FTP server, then you will need to permanently open these ports by adding them to your ASL firewall configuration. Please see the article below for the method thats appropriate for your needs:
https://www.atomicorp.com/wiki/index.php/ASL_firewall
Keep in mind that FTP was never designed to be encrypted, and in some cases when faced with multiple firewalls (including desktop firewalls) it may simply not be possible to pass the FTP data connections through successfully. For this reason, other protocols were designed to replace to FTP, such as SSH and HTTPS.
[edit] proftp example
Note: This is an unsupported example provided as a courtesy. Please contact your FTP vendor for assistance with configuring your FTP server. If you require our assistance, please contact support and we will put a quote together for professional services.
Step 1:
Edit /etc/proftpd.conf, and add the line below:
PassivePorts 60000 60500
Step 2:
Add in the passive ports range to your ASL firewall configuration. For example, if you are using fast mode you would add these ports in this format to your FW_INBOUND_TCP_SERVICES setting:
60000:60050
Note: proftps PassivePorts setting limits the amount of parallel connections and the number of connections (read directory listings and/or file transfers) per net.netfilter.nf_conntrack_tcp_timeout_time_wait seconds. If you will be transfering a lot of small files, you may need to open a wider range of ports. If you are not using the ASL kernel, you will need to contact your kernel vendor for assistance if this does not work for you.
Step 3:
Restart proftp
[edit] Reseting to defaults
To reset the firewall to only those rules generated by ASL, follow these steps:
- iptables-save > /etc/asl/firewall/backup.fw
- service asl-firewall stop
- rm -f /etc/asl/firewall/asl_stop.fw
- service asl-firewall start
[edit] Third party firewall products
ASL is not supported with third party firewall products. You must remove these products, and remove any firewall rules configured on the system by these tools before installing or using ASL.
[edit] Error Messages
Please see the ASL error messages page.
[edit] Additional
[edit] SELinux
SELinux policies have been known to interfere with some RPM updates. This is because SELinux policies are not always adjusted for modern platforms and third party packages, such as control panels. This can manifest itself in mysterious failures in %pre and %post macros (confirmed on RHEL4).
ASL includes an advanced RBAC system that is more powerful and easier to use than SELinux and we recommend you use that instead of SELinux. However, if you wish to use SELinux ASL will work fine with SELinux, however you may need to adjust your SELinux policies for your systems specific needs.
If you encounter any issues with rpm installations on your system, and you are not qualified to adjust your SELinux policies that came with your operating system, we recommend you disable SELinux and use the built in RBAC in ASL.
To disable SELinux set:
selinux=0
in the kernel boot parameters for your system.
setenable 0, setenforce 0 and disabling SELinux with sysctl are not effective. To disable selinux you must boot with selinux=0 set for your system.
