Difference between revisions of "Compromised System"
m |
|||
| (16 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| − | + | '''Compromised System checklist''' | |
| − | + | ||
| + | == Abstract: == | ||
| + | |||
The following is a checklist of tasks to perform when a hosting system has been compromised, to ensure you have all the appropriate data to recover the system and ensure that it will not be compromised again. A key to rapid recovery is to use ASL to minimize the forensic investigation time required to recover. Ideally the specific exploits should be identified in advance, however given time constraints this might not be possible until later. The goal of this checklist is Rapid Recovery. | The following is a checklist of tasks to perform when a hosting system has been compromised, to ensure you have all the appropriate data to recover the system and ensure that it will not be compromised again. A key to rapid recovery is to use ASL to minimize the forensic investigation time required to recover. Ideally the specific exploits should be identified in advance, however given time constraints this might not be possible until later. The goal of this checklist is Rapid Recovery. | ||
| − | + | ||
| + | == Preqreqs: == | ||
| + | |||
1 Backup server, to store 2 copies of data from the compromised system | 1 Backup server, to store 2 copies of data from the compromised system | ||
| Line 18: | Line 22: | ||
| − | + | ||
| + | == Step 1) Find out how the system was compromised == | ||
| + | |||
| + | First determine the level of compromise. Is this a full compromise, or just an individual site. | ||
| + | |||
| + | Task 1) Start with your desktop and the desktop of anyone who accesses your system(s) as a privileged user. About 1/3 of all our forensic cases originate here. Of that 1/3, most of the time users believe that the desktop has not been compromised, and unfortunately most of the time it is! | ||
| + | |||
| + | Please dont do that to yourself, time is of the essence, assume the desktop has been compromised. Its the easiest way to compromise a system, just steal the credentials on the desktop and log in! | ||
| + | |||
| + | Task 2) general rootkit detection (note these tools are LIMITED. They are best used for initial inspections, they will miss a lot) | ||
Rkhunter: | Rkhunter: | ||
| Line 26: | Line 39: | ||
Chkrootkit | Chkrootkit | ||
chkrootkit | chkrootkit | ||
| + | |||
| + | Task 3) look for suspicious processes | ||
| + | |||
| + | Task 4) look for suspicious files | ||
| + | |||
| + | Task 5) create snapshots of memory | ||
| + | |||
| + | Task 6) Boot system from trusted media (CD, PXE, etc) | ||
| + | |||
| + | Task 7) run *trusted* versions of chkrootkit and rkhunter against compromised drive. | ||
| + | |||
| + | Task 8) investigate logfiles | ||
| + | /var/log/messages | ||
| + | /var/log/secure | ||
| + | |||
| + | Task 9) [[Compromised System: FTP]] | ||
| + | |||
| + | Task 10) Confirm that your backups have not been compromised. Dont restore from a backup until you know you can trust it. | ||
| + | |||
| + | Task 11) Image the compromised system if you can, but don't trust anything it tells you. | ||
| + | |||
| + | == '''Step 2) Back up data from the compromised host. We make 2 copies''' == | ||
| + | |||
| + | |||
| + | Task 1: Rsync back of compromised host from the backup server (it is because migration tools amost always miss something. This task will give you a complete copy of the old system) | ||
| + | rsync -av -e ssh root@<IP>:/ /var/backups/<IP>/ | ||
| + | |||
| + | Task 1: On the compromised host, create a Plesk Backup | ||
| + | mkdir /root/backups | ||
| + | |||
| + | PSA 7.5 and lower | ||
| + | /usr/loca/psa/bin/psadump -f | split -b1000m /root/backups/backup. | ||
| + | |||
| + | PSA 8.0 and higher | ||
| + | /usr/local/psa/bin/pleskbackup all --split=1G /root/backups/backup | ||
| + | |||
| + | |||
| + | Task 2: Rsync back of compromised host from the backup server (this gets those backups too): | ||
| + | rsync -av -e ssh root@<IP>:/ /var/backups/<IP>/ | ||
| + | |||
| + | |||
| + | |||
| + | |||
| + | == '''Step 3) Reinstall the system''' == | ||
| + | |||
| + | |||
| + | Task 1: Reimage the system | ||
| + | |||
| + | Optional: The AOOI script to image the system with CentOS 4 or 5 (1and1 users, or users on other EOL'd operating systems like FC4, FC5, etc) | ||
| + | wget -q -O - https://www.atomicorp.com/installer/aooi |sh | ||
| − | + | Task 2: Update the system | |
| + | |||
| + | yum -y update | ||
| − | + | '''Step 4) Install/Configure Atomic Secured Linux''' | |
| − | + | ||
| − | Task | + | Task 1: Install ASL |
| − | + | ||
| + | wget -q -O - https://www.atomicorp.com/installers/asl |sh | ||
| − | Task | + | Task 2: Update signatures |
| + | |||
| + | aum -u | ||
| + | Task 3: Run ASL in fix mode | ||
| + | |||
| + | asl -s -f | ||
| − | + | Task 4: Install Plesk (yum or autoupdater) | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | Using Yum: | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | sub-task 1: Configure PSA channel for the version of your backup you made (ie, psa 7.5 backup, install psa 7.5) | |
| − | + | ||
| − | + | See http://www.atomicorp.com/channels/plesk/ for plesk channels | |
| − | + | ||
| − | + | Example setting up PSA channel using the atomic installer: | |
| + | wget -q -O - https://www.atomicorp.com/installer/atomic |sh | ||
| − | Step 5) Restore | + | |
| − | + | sub-task 2: Install psa, and support packages | |
| − | + | yum -y install psa psa-bu mailman psa-spamassassin frontpage | |
| − | Task 2: Use psarestore/pleskrestore to recover data | + | |
| − | + | sub-task 3: copy psa.key from rsync backup on the backup server to /etc/psa/psa.key on the new system | |
| + | scp /backup/<IP>/etc/psa/psa.key root@<IP>:/etc/psa/psa.key | ||
| + | |||
| + | sub-task 4: restart psa | ||
| + | /etc/init.d/psa restart | ||
| + | |||
| + | sub-task 5: log into psa, and reconfigure settings. Specifically set the shared IP's | ||
| + | https://<IP>:8443 | ||
| + | |||
| + | == '''Step 5) Restore system''' == | ||
| + | |||
| + | |||
| + | Task 1: Copy plesk backup to reimaged system | ||
| + | scp /var/backups/<IP>/root/backups/* root@<IP>:/root/ | ||
| + | |||
| + | Task 2: Use psarestore/pleskrestore to recover data | ||
| + | /usr/local/psa/bin/pleskrestore | ||
| − | + | '''Step 6) Restore additional Components''' | |
| − | + | ||
Latest revision as of 13:06, 20 February 2013
Compromised System checklist
Contents |
[edit] Abstract:
The following is a checklist of tasks to perform when a hosting system has been compromised, to ensure you have all the appropriate data to recover the system and ensure that it will not be compromised again. A key to rapid recovery is to use ASL to minimize the forensic investigation time required to recover. Ideally the specific exploits should be identified in advance, however given time constraints this might not be possible until later. The goal of this checklist is Rapid Recovery.
[edit] Preqreqs:
1 Backup server, to store 2 copies of data from the compromised system
1 Valid ASL subscription
Optional: Serial port/KVM console access
Optional: Rescue mode PXE image
[edit] Step 1) Find out how the system was compromised
First determine the level of compromise. Is this a full compromise, or just an individual site.
Task 1) Start with your desktop and the desktop of anyone who accesses your system(s) as a privileged user. About 1/3 of all our forensic cases originate here. Of that 1/3, most of the time users believe that the desktop has not been compromised, and unfortunately most of the time it is!
Please dont do that to yourself, time is of the essence, assume the desktop has been compromised. Its the easiest way to compromise a system, just steal the credentials on the desktop and log in!
Task 2) general rootkit detection (note these tools are LIMITED. They are best used for initial inspections, they will miss a lot)
Rkhunter:
rkhunter --update rkhunter -c -sk
Chkrootkit
chkrootkit
Task 3) look for suspicious processes
Task 4) look for suspicious files
Task 5) create snapshots of memory
Task 6) Boot system from trusted media (CD, PXE, etc)
Task 7) run *trusted* versions of chkrootkit and rkhunter against compromised drive.
Task 8) investigate logfiles
/var/log/messages /var/log/secure
Task 9) Compromised System: FTP
Task 10) Confirm that your backups have not been compromised. Dont restore from a backup until you know you can trust it.
Task 11) Image the compromised system if you can, but don't trust anything it tells you.
[edit] Step 2) Back up data from the compromised host. We make 2 copies
Task 1: Rsync back of compromised host from the backup server (it is because migration tools amost always miss something. This task will give you a complete copy of the old system)
rsync -av -e ssh root@<IP>:/ /var/backups/<IP>/
Task 1: On the compromised host, create a Plesk Backup
mkdir /root/backups
PSA 7.5 and lower
/usr/loca/psa/bin/psadump -f | split -b1000m /root/backups/backup.
PSA 8.0 and higher
/usr/local/psa/bin/pleskbackup all --split=1G /root/backups/backup
Task 2: Rsync back of compromised host from the backup server (this gets those backups too):
rsync -av -e ssh root@<IP>:/ /var/backups/<IP>/
[edit] Step 3) Reinstall the system
Task 1: Reimage the system
Optional: The AOOI script to image the system with CentOS 4 or 5 (1and1 users, or users on other EOL'd operating systems like FC4, FC5, etc)
wget -q -O - https://www.atomicorp.com/installer/aooi |sh
Task 2: Update the system
yum -y update
Step 4) Install/Configure Atomic Secured Linux
Task 1: Install ASL
wget -q -O - https://www.atomicorp.com/installers/asl |sh
Task 2: Update signatures
aum -u
Task 3: Run ASL in fix mode
asl -s -f
Task 4: Install Plesk (yum or autoupdater)
Using Yum:
sub-task 1: Configure PSA channel for the version of your backup you made (ie, psa 7.5 backup, install psa 7.5)
See http://www.atomicorp.com/channels/plesk/ for plesk channels
Example setting up PSA channel using the atomic installer:
wget -q -O - https://www.atomicorp.com/installer/atomic |sh
sub-task 2: Install psa, and support packages
yum -y install psa psa-bu mailman psa-spamassassin frontpage
sub-task 3: copy psa.key from rsync backup on the backup server to /etc/psa/psa.key on the new system
scp /backup/<IP>/etc/psa/psa.key root@<IP>:/etc/psa/psa.key
sub-task 4: restart psa
/etc/init.d/psa restart
sub-task 5: log into psa, and reconfigure settings. Specifically set the shared IP's
https://<IP>:8443
[edit] Step 5) Restore system
Task 1: Copy plesk backup to reimaged system
scp /var/backups/<IP>/root/backups/* root@<IP>:/root/
Task 2: Use psarestore/pleskrestore to recover data
/usr/local/psa/bin/pleskrestore
Step 6) Restore additional Components
