Linode Forum
Linode Community Forums 		 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Post new topic  Reply to topic

Board index » Linux Community Forums » General Discussion
  Previous topic | Next topic 
Author Message
spamoom
PostPosted: Tue Apr 05, 2011 5:57 pm 
Offline
Senior Newbie

Joined: Tue Apr 05, 2011 5:46 pm
Posts: 7
Hi All!

I've been a linode customer for almost a year now, but have recently been made aware that my server has been accessing another website maliciously. I'm told that I'm going to have to redeploy (really not an option) unless I can locate and fix the problem.

I'd say I am at an intermediate skill level with linux so I'm not really sure what else I can do apart from searching the entire file system for the domain name and ip (which returned no results)

root@server [/]# find . | xargs grep 'THE_IP' -sl
root@server [/]# find . | xargs grep 'THE_DOMAIN' -sl

I'd really appreciate any help with how I can find out what could have gone wrong!!

Here are the logs on the server that is being attacked by mine...

Access Log:
Code:
MY_IP - - [05/Apr/2011:12:43:53 -0500] "GET /index.php?cPath=35/admin/file_manager.php/login.php HTTP/1.1" 200 19334 "-" "Mozilla/3.0 (X11; I; SunOS 5.4 sun4m)"
MY_IP - - [05/Apr/2011:12:43:54 -0500] "GET /admin/file_manager.php/login.php HTTP/1.1" 404 8802 "-" "Mozilla/3.0 (X11; I; SunOS 5.4 sun4m)"
MY_IP - - [05/Apr/2011:12:43:54 -0500] "GET /index.php?cPath=35/admin/categories.php/login.php?cPath=&action=new_product_preview HTTP/1.1" 200 43381 "-" "Mozilla/3.0 (X11; I; SunOS 5.4 sun4m)"
MY_IP - - [05/Apr/2011:12:43:55 -0500] "GET /admin/categories.php/login.php?cPath=&action=new_product_preview HTTP/1.1" 404 8802 "-" "Mozilla/3.0 (X11; I; SunOS 5.4 sun4m)"


Error Log:
Code:
[Tue Apr 05 12:43:53 2011] [error] [client MY_IP] ModSecurity: Warning. Match of "rx ^OPTIONS$" against "REQUEST_METHOD" required. [file "/etc/httpd/modsecurity.d/modsecurity_crs_21_protocol_anomalies.conf"] [line "41"] [id "960015"] [msg "Request Missing an Accept Header"] [severity "CRITICAL"] [tag "PROTOCOL_VIOLATION/MISSING_HEADER"] [hostname "THEIR_DOMAIN"] [uri "/index.php"] [unique_id "TZtU2Uj5jcIAAAqtDHMAAAA3"]
[Tue Apr 05 12:43:54 2011] [error] [client MY_IP] ModSecurity: Warning. Match of "rx ^OPTIONS$" against "REQUEST_METHOD" required. [file "/etc/httpd/modsecurity.d/modsecurity_crs_21_protocol_anomalies.conf"] [line "41"] [id "960015"] [msg "Request Missing an Accept Header"] [severity "CRITICAL"] [tag "PROTOCOL_VIOLATION/MISSING_HEADER"] [hostname "THEIR_DOMAIN"] [uri "/admin/file_manager.php/login.php"] [unique_id "TZtU2kj5jcIAAAZjgU4AAAAc"]
[Tue Apr 05 12:43:54 2011] [error] [client MY_IP] File does not exist: /home/kdc/public_html/admin
[Tue Apr 05 12:43:54 2011] [error] [client MY_IP] ModSecurity: Warning. Match of "rx ^OPTIONS$" against "REQUEST_METHOD" required. [file "/etc/httpd/modsecurity.d/modsecurity_crs_21_protocol_anomalies.conf"] [line "41"] [id "960015"] [msg "Request Missing an Accept Header"] [severity "CRITICAL"] [tag "PROTOCOL_VIOLATION/MISSING_HEADER"] [hostname "THEIR_DOMAIN"] [uri "/index.php"] [unique_id "TZtU2kj5jcIAAAuqFkwAAABA"]
[Tue Apr 05 12:43:55 2011] [error] [client MY_IP] ModSecurity: Warning. Match of "rx ^OPTIONS$" against "REQUEST_METHOD" required. [file "/etc/httpd/modsecurity.d/modsecurity_crs_21_protocol_anomalies.conf"] [line "41"] [id "960015"] [msg "Request Missing an Accept Header"] [severity "CRITICAL"] [tag "PROTOCOL_VIOLATION/MISSING_HEADER"] [hostname "THEIR_DOMAIN"] [uri "/admin/categories.php/login.php"] [unique_id "TZtU20j5jcIAAF9IsRAAAAAE"]
[Tue Apr 05 12:43:55 2011] [error] [client MY_IP] File does not exist: /home/kdc/public_html/admin
Top
   
Reply with quote  
haus
 Post subject:
PostPosted: Tue Apr 05, 2011 6:21 pm 
Offline
Senior Member

Joined: Wed Mar 03, 2010 2:04 pm
Posts: 111
It may not be a hard-coded script, it might be a proxy server has been set up on your linode and someone is directing the attack from another location through your linode.

I wonder if someone is accessing your linode at the exact same times as the remote logs you posted.

When I typed that user-agent from the logs into google, one of the results came up with something about Net::Proxy. Not sure if it's related, but it is one place to look if you're not finding a hard-coded script anywhere on your box.
Top
   
Reply with quote  
eld101
 Post subject:
PostPosted: Tue Apr 05, 2011 7:05 pm 
Offline
Senior Member

Joined: Mon Sep 01, 2008 5:14 pm
Posts: 92
Have you tried installing http://www.chkrootkit.org/?
Top
   
Reply with quote  
spamoom
 Post subject:
PostPosted: Wed Apr 06, 2011 3:51 am 
Offline
Senior Newbie

Joined: Tue Apr 05, 2011 5:46 pm
Posts: 7
No luck with Net::Proxy or User agent search...

This stands out from chkrootkit!

Checking `bindshell'... INFECTED (PORTS: 465)
Top
   
Reply with quote  
obs
 Post subject:
PostPosted: Wed Apr 06, 2011 4:52 am 
Offline
Senior Member

Joined: Sun Mar 07, 2010 7:47 pm
Posts: 1970
Website: http://www.rwky.net
Location: Earth
The bind shell one is possibly a false positive see http://www.chkrootkit.org/faq/#7

What does netstat -lpntu show?

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue
Top
   
Reply with quote  
spamoom
 Post subject:
PostPosted: Wed Apr 06, 2011 4:56 am 
Offline
Senior Newbie

Joined: Tue Apr 05, 2011 5:46 pm
Posts: 7
It shows this :) (The server is running WHM/cPanel)

Thanks for all your help - much appreciated!

Code:
root@server1 [~]# netstat -lpntu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:993                 0.0.0.0:*                   LISTEN      2996/dovecot        
tcp        0      0 0.0.0.0:2082                0.0.0.0:*                   LISTEN      22406/cpsrvd - wait 
tcp        0      0 0.0.0.0:2083                0.0.0.0:*                   LISTEN      22406/cpsrvd - wait 
tcp        0      0 0.0.0.0:995                 0.0.0.0:*                   LISTEN      2996/dovecot        
tcp        0      0 0.0.0.0:2086                0.0.0.0:*                   LISTEN      22406/cpsrvd - wait 
tcp        0      0 0.0.0.0:2087                0.0.0.0:*                   LISTEN      22406/cpsrvd - wait 
tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      3388/mysqld         
tcp        0      0 0.0.0.0:110                 0.0.0.0:*                   LISTEN      2996/dovecot        
tcp        0      0 0.0.0.0:2095                0.0.0.0:*                   LISTEN      22406/cpsrvd - wait 
tcp        0      0 127.0.0.1:783               0.0.0.0:*                   LISTEN      15006/spamd child   
tcp        0      0 0.0.0.0:143                 0.0.0.0:*                   LISTEN      2996/dovecot        
tcp        0      0 0.0.0.0:2096                0.0.0.0:*                   LISTEN      22406/cpsrvd - wait 
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      534/httpd           
tcp        0      0 0.0.0.0:465                 0.0.0.0:*                   LISTEN      15574/exim          
tcp        0      0 0.0.0.0:21                  0.0.0.0:*                   LISTEN      15535/pure-ftpd (SE 
tcp        0      0 178.79.154.28:53            0.0.0.0:*                   LISTEN      2795/named          
tcp        0      0 178.79.145.220:53           0.0.0.0:*                   LISTEN      2795/named          
tcp        0      0 127.0.0.1:53                0.0.0.0:*                   LISTEN      2795/named          
tcp        0      0 0.0.0.0:25                  0.0.0.0:*                   LISTEN      15574/exim          
tcp        0      0 127.0.0.1:953               0.0.0.0:*                   LISTEN      2795/named          
tcp        0      0 0.0.0.0:443                 0.0.0.0:*                   LISTEN      534/httpd           
tcp        0      0 0.0.0.0:2077                0.0.0.0:*                   LISTEN      22400/cpdavd - acce 
tcp        0      0 0.0.0.0:2078                0.0.0.0:*                   LISTEN      22400/cpdavd - acce 
tcp        0      0 :::465                      :::*                        LISTEN      15574/exim          
tcp        0      0 :::21                       :::*                        LISTEN      15535/pure-ftpd (SE 
tcp        0      0 :::22                       :::*                        LISTEN      2815/sshd           
tcp        0      0 :::25                       :::*                        LISTEN      15574/exim          
udp        0      0 178.79.154.28:53            0.0.0.0:*                               2795/named          
udp        0      0 178.79.145.220:53           0.0.0.0:*                               2795/named          
udp        0      0 127.0.0.1:53                0.0.0.0:*                               2795/named          
udp        0      0 0.0.0.0:68                  0.0.0.0:*                               2662/dhclient       
udp        0      0 178.79.154.28:123           0.0.0.0:*                               2829/ntpd           
udp        0      0 178.79.145.220:123          0.0.0.0:*                               2829/ntpd           
udp        0      0 127.0.0.1:123               0.0.0.0:*                               2829/ntpd           
udp        0      0 0.0.0.0:123                 0.0.0.0:*                               2829/ntpd           
udp        0      0 ::1:123                     :::*                                    2829/ntpd           
udp        0      0 :::123                      :::*                                    2829/ntpd
Top
   
Reply with quote  
obs
 Post subject:
PostPosted: Wed Apr 06, 2011 7:58 am 
Offline
Senior Member

Joined: Sun Mar 07, 2010 7:47 pm
Posts: 1970
Website: http://www.rwky.net
Location: Earth
If it's running whm then yes it is a false positive.

However there are a few "problems" you'll want to address.

1) SSH has password authentication enabled
2) MySQL is listening on the public interface
3) You're using apache 2.0 you should really be using 2.2 by now.
4) You have front page extensions enabled these are no longer supported by microsoft.

Have a check in /var/log/auth.log for unauthorized access attempts via ssh.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue
Top
   
Reply with quote  
spamoom
 Post subject:
PostPosted: Wed Apr 06, 2011 8:22 am 
Offline
Senior Newbie

Joined: Tue Apr 05, 2011 5:46 pm
Posts: 7
Great, I've fixed all those problems (apache 2.2 still compiling)

/var/log/auth.log doesn't exist - so can't see what attempts were made :s

I have however found that a bunch of perl scripts were eating my CPU and file IO owned by a website which I have found to be hacked into and malicious scripts had been uploaded - thinking this could have been the proxy to which the hackers were using my server!

Thanks again obs, eld101 and haus :)
Top
   
Reply with quote  
JshWright
 Post subject:
PostPosted: Wed Apr 06, 2011 9:41 am 
Offline
Senior Member

Joined: Mon Oct 27, 2008 10:24 am
Posts: 173
Website: http://www.worshiproot.com
Wait... you have to compile Apache 2.2?

What distro (and version) are you running?
Top
   
Reply with quote  
spamoom
 Post subject:
PostPosted: Wed Apr 06, 2011 9:49 am 
Offline
Senior Newbie

Joined: Tue Apr 05, 2011 5:46 pm
Posts: 7
Nothing serious, WHM has a module called easyapache which does it all for me :) Compiles against the apache modules rather than having them included at runtime.

CENTOS 5.5 ;)

http://docs.cpanel.net/twiki/bin/view/EasyApache3/
Top
   
Reply with quote  
obs
 Post subject:
PostPosted: Wed Apr 06, 2011 10:22 am 
Offline
Senior Member

Joined: Sun Mar 07, 2010 7:47 pm
Posts: 1970
Website: http://www.rwky.net
Location: Earth
If you've a hacked site then yes that's probably the cause of the problem, suspend the site and do further investigation of the box, cpanel has a tendency of changing the system quite heavily, there will be (or should be) some form of log in /var/log which contains ssh attempts.

You should also inspect the malicious perl scripts to see if they managed to edit any files, do you know what user the scripts were running under?

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue
Top
   
Reply with quote  
spamoom
 Post subject:
PostPosted: Wed Apr 06, 2011 11:29 am 
Offline
Senior Newbie

Joined: Tue Apr 05, 2011 5:46 pm
Posts: 7
Yes I do know which user was running the perl scripts, that's what helped me find out which site was running the processes :)
Top
   
Reply with quote  
haus
 Post subject:
PostPosted: Wed Apr 06, 2011 12:00 pm 
Offline
Senior Member

Joined: Wed Mar 03, 2010 2:04 pm
Posts: 111
Look for the cpanel SSH log on CentOS in /var/log/secure (that's where mine is on another server). If someone got bruteforced you should be using something like CSF/LFD or fail2ban. If they got the password through another means...not much you can do if you're going to allow other users on your box.

I'd love to know more about those scripts they uploaded.

You can also set up an "ftpcheck" script to monitor FTP uploads and look for anything suspicious, though if you have customers uploading files that might go against privacy concerns. The problem now is that once your box is compromised, you never quite know if you've gotten it all out or not. Hopefully the damage was limited to that one user account and a handful of perl scripts.
Top
   
Reply with quote  
obs
 Post subject:
PostPosted: Wed Apr 06, 2011 12:00 pm 
Offline
Senior Member

Joined: Sun Mar 07, 2010 7:47 pm
Posts: 1970
Website: http://www.rwky.net
Location: Earth
I assume that account doesn't have sudo access if it does you'll want to check the logs for sudo uses.

You could also run find / -user username -exec ls -lh {} \;
to list all files owned by the user and their details i.e. last modified etc, so you can see if any new files have been created.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue
Top
   
Reply with quote  
Guspaz
 Post subject:
PostPosted: Wed Apr 06, 2011 5:33 pm 
Offline
Senior Member
User avatar

Joined: Tue May 26, 2009 3:29 pm
Posts: 1691
Location: Montreal, QC
obs wrote:
I assume that account doesn't have sudo access if it does you'll want to check the logs for sudo uses.

You could also run find / -user username -exec ls -lh {} \;
to list all files owned by the user and their details i.e. last modified etc, so you can see if any new files have been created.


Doesn't matter, the logs can be modified. As soon as root is compromised, or a sudoers user is compromised, the box is too compromised to save, and must be rebuilt.
Top
   
Reply with quote  
Display posts from previous:  Sort by  
Post new topic  Reply to topic

Board index » Linux Community Forums » General Discussion


Who is online

Users browsing this forum: No registered users and 6 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group