Hi folks!

I have recently discovered some very suspicious files on my box and I hope there is an expert who may be able to help.

I assume you will want me to post logs or netstat output, but I am not sure what is most relevant, so I just wait until you ask. Here the most basic info:

Server:

Ubuntu 10.04 LTS with ISPConfig 2.2.40
PHP downgraded to PHP 5.2.10-2ubuntu6.10 with Suhosin-Patch 0.9.7
(in order to run a Drupal 5 site)

Problem:
I discovered some very weird files in /var/www/ :

drwxr-xr-x  3 root        root  4096 2011-09-23 00:30 13441:2e4ad885f14b898d2d97464014ee88ff:Trojan.Vundo-32951
drwxr-xr-x  3 root        root  4096 2011-09-23 00:30 64000:909caa0397babc8dbaec55bb804b268d:Worm.Palevo-15930
drwxr-xr-x  3 root        root  4096 2011-09-23 00:30 66560:eae82f3da7fbf68c7d9a21478b29db1f:Worm.Palevo-15927
drwxr-xr-x  3 root        root  4096 2011-09-23 00:30 80384:d12a97d07a024www.paperin.org
drwxr-xr-x  3 root        root  4096 2011-09-23 00:30 86016:1a39b0adeb471b8d5be710b10c8fc4ee:Worm.Palevo-15928
drwxr-xr-x  3 root        root  4096 2011-09-23 00:30 93696:6957c2d714de628defa50c4eb6364e48:Worm.Palevo-15931
drwxr-xr-x  3 root        root  4096 2011-09-23 00:30 95744:9db0c0862577fd8db9ef1d2cd2cd45a5:Worm.Palevo-15929

Note, www DOT paperin DOT org is a one of several small sites hosted on the box. It runs on Drupal 5.
I do not know where these came from; I did not create them knowingly for sure.

I deleted these files, but I am not sure how to proceed.

In case that it is relevant, here is a deep listing of the suspicious directories.

I really appreciate your help!

user@host: /var/www# ls -al

./13441:2e4ad885f14b898d2d97464014ee88ff:Trojan.Vundo-32951
:
total 4
drwxr-xr-x 3 root root 4096 2011-09-23 00:30 log

./13441:2e4ad885f14b898d2d97464014ee88ff:Trojan.Vundo-32951
/log:
total 8
drwxr-xr-x 3 root root 4096 2011-09-23 00:30 2011
lrwxrwxrwx 1 root root   87 2011-09-23 00:30 web.log -> /var/www/13441:2e4ad885f14b898d2d97464014ee88ff:Trojan.Vundo-32951
/log/2011/09/web.log

./13441:2e4ad885f14b898d2d97464014ee88ff:Trojan.Vundo-32951
/log/2011:
total 4
drwxr-xr-x 2 root root 4096 2011-09-23 00:30 09

./13441:2e4ad885f14b898d2d97464014ee88ff:Trojan.Vundo-32951
/log/2011/09:
total 0
-rw-r--r-- 1 root root 0 2011-09-23 00:30 web.log

./64000:909caa0397babc8dbaec55bb804b268d:Worm.Palevo-15930
:
total 4
drwxr-xr-x 3 root root 4096 2011-09-23 00:30 log

./64000:909caa0397babc8dbaec55bb804b268d:Worm.Palevo-15930
/log:
total 8
drwxr-xr-x 3 root root 4096 2011-09-23 00:30 2011
lrwxrwxrwx 1 root root   86 2011-09-23 00:30 web.log -> /var/www/64000:909caa0397babc8dbaec55bb804b268d:Worm.Palevo-15930
/log/2011/09/web.log

./64000:909caa0397babc8dbaec55bb804b268d:Worm.Palevo-15930
/log/2011:
total 4
drwxr-xr-x 2 root root 4096 2011-09-23 00:30 09

./64000:909caa0397babc8dbaec55bb804b268d:Worm.Palevo-15930
/log/2011/09:
total 0
-rw-r--r-- 1 root root 0 2011-09-23 00:30 web.log

./66560:eae82f3da7fbf68c7d9a21478b29db1f:Worm.Palevo-15927
:
total 4
drwxr-xr-x 3 root root 4096 2011-09-23 00:30 log

./66560:eae82f3da7fbf68c7d9a21478b29db1f:Worm.Palevo-15927
/log:
total 8
drwxr-xr-x 3 root root 4096 2011-09-23 00:30 2011
lrwxrwxrwx 1 root root   86 2011-09-23 00:30 web.log -> /var/www/66560:eae82f3da7fbf68c7d9a21478b29db1f:Worm.Palevo-15927
/log/2011/09/web.log

./66560:eae82f3da7fbf68c7d9a21478b29db1f:Worm.Palevo-15927
/log/2011:
total 4
drwxr-xr-x 2 root root 4096 2011-09-23 00:30 09

./66560:eae82f3da7fbf68c7d9a21478b29db1f:Worm.Palevo-15927
/log/2011/09:
total 0
-rw-r--r-- 1 root root 0 2011-09-23 00:30 web.log

./80384:d12a97d07a024www.paperin.org:
total 4
drwxr-xr-x 3 root root 4096 2011-09-23 00:30 log

./80384:d12a97d07a024www.paperin.org/log:
total 4
drwxr-xr-x 3 root root 4096 2011-09-23 00:30 2011

./80384:d12a97d07a024www.paperin.org/log/2011:
total 4
drwxr-xr-x 2 root root 4096 2011-09-23 00:30 09

./80384:d12a97d07a024www.paperin.org/log/2011/09:
total 4
-rw-r--r-- 1 root root 167 2011-09-23 00:30 web.log

./86016:1a39b0adeb471b8d5be710b10c8fc4ee:Worm.Palevo-15928
:
total 4
drwxr-xr-x 3 root root 4096 2011-09-23 00:30 log

./86016:1a39b0adeb471b8d5be710b10c8fc4ee:Worm.Palevo-15928
/log:
total 8
drwxr-xr-x 3 root root 4096 2011-09-23 00:30 2011
lrwxrwxrwx 1 root root   86 2011-09-23 00:30 web.log -> /var/www/86016:1a39b0adeb471b8d5be710b10c8fc4ee:Worm.Palevo-15928
/log/2011/09/web.log

./86016:1a39b0adeb471b8d5be710b10c8fc4ee:Worm.Palevo-15928
/log/2011:
total 4
drwxr-xr-x 2 root root 4096 2011-09-23 00:30 09

./86016:1a39b0adeb471b8d5be710b10c8fc4ee:Worm.Palevo-15928
/log/2011/09:
total 0
-rw-r--r-- 1 root root 0 2011-09-23 00:30 web.log

./93696:6957c2d714de628defa50c4eb6364e48:Worm.Palevo-15931
:
total 4
drwxr-xr-x 3 root root 4096 2011-09-23 00:30 log

./93696:6957c2d714de628defa50c4eb6364e48:Worm.Palevo-15931
/log:
total 8
drwxr-xr-x 3 root root 4096 2011-09-23 00:30 2011
lrwxrwxrwx 1 root root   86 2011-09-23 00:30 web.log -> /var/www/93696:6957c2d714de628defa50c4eb6364e48:Worm.Palevo-15931
/log/2011/09/web.log

./93696:6957c2d714de628defa50c4eb6364e48:Worm.Palevo-15931
/log/2011:
total 4
drwxr-xr-x 2 root root 4096 2011-09-23 00:30 09

./93696:6957c2d714de628defa50c4eb6364e48:Worm.Palevo-15931
/log/2011/09:
total 0
-rw-r--r-- 1 root root 0 2011-09-23 00:30 web.log

./95744:9db0c0862577fd8db9ef1d2cd2cd45a5:Worm.Palevo-15929
:
total 4
drwxr-xr-x 3 root root 4096 2011-09-23 00:30 log

./95744:9db0c0862577fd8db9ef1d2cd2cd45a5:Worm.Palevo-15929
/log:
total 8
drwxr-xr-x 3 root root 4096 2011-09-23 00:30 2011
lrwxrwxrwx 1 root root   86 2011-09-23 00:30 web.log -> /var/www/95744:9db0c0862577fd8db9ef1d2cd2cd45a5:Worm.Palevo-15929
/log/2011/09/web.log

./95744:9db0c0862577fd8db9ef1d2cd2cd45a5:Worm.Palevo-15929
/log/2011:
total 4
drwxr-xr-x 2 root root 4096 2011-09-23 00:30 09

./95744:9db0c0862577fd8db9ef1d2cd2cd45a5:Worm.Palevo-15929
/log/2011/09:
total 0
-rw-r--r-- 1 root root 0 2011-09-23 00:30 web.log

...other (expected) directories follow...

Those directory names look a little too obviously sketchy... I'd expect something more subtle, you know what I mean?

It might be worth checking with the ISPConfig folks to see what might have created those directories... it was something running as root, so it probably wasn't a simple PHP exploit (although you are running a likely-exploitable version of PHP).

Whenever you discover any suspicious files, it is a good idea to make a tarball of the suspicious files before deleting them. That way, if some sort of exploit is suspected, those files can help investigate. Run:

tar cjvf suspcious.tar.bz2 /path/to/suspicious/files

You can specify a directory file here. That will generate suspicious.tar.bz2 containing your files in whichever directory and place it in the directory you run that command from. After doing that, you can remove the files.

_________________
Kris the Piki Geeker

Also, remove that tarball from the machine you found them on. For example, you can move the tarball off of your Linode and onto your home computer, then from there burn the tarball to a CD or put it on a flash drive, then delete it from your home computer.

It is also good to check your logs -- ssh logs, ftp logs (if you have a ftp server), and any logs pertaining to your web servers (apache, php, etc). If you have anything else on your Linode that is available to the outside world, examine those logs too. Look for anything that looks suspicious. Make a copy of any log that contains suspicious information so that suspicious info doesn't get overwritten during any further logging. Some programs will occasionally rename the log and start a new one, e.g. rename /var/log/example.log to /var/log/example.log.1 and create an empty /var/log/example.log, to make the logs smaller and easier to read, and to have the old logs available for situations like this, but it's a good idea to make a copy just in case.

_________________
Kris the Piki Geeker

Of course, if your system *was* compromised, the first thing the attacker is going to do is erase anything suspicious from the logs.

_________________
Matt Nordhoff (aka Peng on IRC)

It is possible for the attacker to forget to do this, especially a newbie attacker, or someone who recently started using Linux. Also, every distro is different, while most use /var/log for this, the exact layout for /var/log can be different, and it is possible for the user to move these logs via config files for their software. It is possible to have "separated logs" for things like apache, so I'd have my main apache log in /var/log (or wherever I decide to tell apache to hide it), and individual access and error logs for each of my vhosts, each hidden in other directories. With correct permissions on these logs and good security setup, even if my web sites allows people to upload stuff, the logs will still be well hidden. I can also make it very hard to access root -- disable root logins and passwords in my ssh daemon, and force the use of RSA keys, and as long as my private key is passworded and I guard it carefully, the attacker would need to find another exploit, which could be made difficult with proper permissions and iptables, and with well-written php hosted on my server.

_________________
Kris the Piki Geeker

Nevertheless, it is good to check logs just in case, make the tarball of the suspicious files and move it off site, and to also check to see if your config files changed (which I forgot to mention in my previous post, config files could possible be changed to make future exploits easier).

_________________
Kris the Piki Geeker

Thanks everyone!

The key points are:

• I have deleted the files without archiving them first. Should they re-appear I will make sure to make a copy first.
• I went though the logs before posting here and did not find anything concerning (which made me even more suspicious). I will look again when I get to a place with access tonight.
• What should I do next? Would anyone has any advice on what steps to take. Should I do nothing? Should I take some pre-emptive action? Where did these weird directories with names that include "Trojan.Vundo" and "Worm.Palevo" come from?

If people are not sure, can anyone recommend a forum where folks may know about this?

Thanks a lot!!

-Start fresh with a new linode
-Check, double check, triple check your sites for compromised code. I'd recommend running through an IDS of sorts in the future.
-NEVER, EVER, EVER DOWNGRADE--EVER. For any reason. If the application CANNOT WORK with upgraded software, ditch it for someone who keeps updated.
-Run something like Modsecurity
-For every site on your linode, change their database passwords, update the site configs
-For any 'users' of any forums or login systems on any of your sites, require them to change their passwords.
-Announce that you've found a compromise and are dealing with it and working to not have one again.
-Read up on websecurity. OWASP has great information on this.

Thanks A-KO.

I really appreciate your advise. However, it is not practical in some details:

• Some applications may require an older runtime version (PHP, Java, .NET, a library, whatever it is). I am very enthusiastic about running the most up-to-date version, but from the point of view of businesses that use applications, this is the reality.
• Rebuilding the server is very time consuming. Of course, I would do that if I knew what went wrong. But without understanding the problem, I might invest a lot of time just to run into trouble again. I would like to understand the issue first, before deciding whether it warrants taking the server down and how I can make sure it does not happen again.

Any ideas on how to investigate what is going on?

Cheers!

Easy: Out of date software. Typically if you're not applying security patches or the latest security releases then you're going to run into trouble.

There are plenty of ways to do web app penetration testing and I'm not really inclined to go into the massive detail in this thread right now, as it's not the time and place.

I would recommend running something like Nessus or Metasploit, or pay for a person or company to do it for you to find vulnerabilities and information disclosure in your system.

Back to the software point at hand, you quoted Microsoft software in a forum regarding Linux. Not that I'm getting on you for that, but the fact that the patch release strategy is typically entirely different.

For example, PHP.NET no longer supports PHP 5.2.x, which means all security flaws found in it will no longer be patched. Any new security flaws found will not be provided with upstream fixes. You're free to patch whatever you can, but you run that risk.

A business running this software will typically have an IDS or IPS which will detect attacks in real time. So they have an extra level of security that looks for common/used exploits like those found in Metasploit--realtime, as the network passes through (look at SNORT).

Again, you have to consider the release cycle of each piece of software you use, and certain distributions have different levels of support and release cycles, and they typically work around some of the upstream providers. You need to be very proactive.

Other things you can look at are MODSECURITY, a form of IDS/IPS that operates directly in Apache and helps guard against the most common of exploits and disclosures. For example, it will help guard against common SQL Injection attacks, information leakage, SQL queries that try to pick from the mysql.users table, amongst many other attempts at exploits. It's very difficult to work with unless you know REGEX really well.

Apache's default logs will not include POST requests, even on the most verbose settings--which is how many attacks come in. I typically use mod_security (though there are other logging mechanisms) that can do full logs of the entire request from start to finish. You see every bit of data that was sent and received from the server. The log files can climb fast, I typically push a few hundred MB of logs/month even on a small website. Thankfully, text compresses really well.

Then run an older LTS-style release that has the right major revision of the software that you need with the latest backported security fixes. If you need software even older than this, you're asking for trouble, because you're going to be open to all sorts of security holes, and whatever you need to do is probably not going to work all that far into the future with dead-end software.



"time consuming" should never come into the equation when figuring out if you've been badly compromised enough to warrant a rebuild. You either have, and must rebuild, or haven't and don't need to.

Very true. I am looking to figure out exactly what happens before deleting and re-building the server.

If you absolutely require PHP 5.2, Ubuntu 8.04 LTS has 5.2.4, with security fixes backported, and it will be supported for another 18 months or so. There is also Drupal 5.7 in hardy, but it is not in the main repository and may not get much support going forward.

From drupal.org,



This pretty much means that you are going to have to, at least once a year, do whatever is necessary to migrate to a new major release of Drupal, or you're going to have Problems. If you haven't done that this year, you're probably having Problems. It's like replacing furnace filters or changing engine oil.

Lucky my furnace hasn't caught fire... Though it does break down twice a year with the change in seasons, which can and will happen on a server of any kind without proper maintenance. Hands down and fists banging on the table, keeping up-to-date is one of the three most important things you can do to reduce your chances of compromise -- if there is a newer version of your software that contains a security fix (and there undoubtedly will be), you should definitely update regardless of what it takes.

Another important part of securing your server's security is checking over your server's firewall. The best firewall policy is to allow only what you absolutely need to and to block everything else. If it doesn't need to be accessible to the outside world, don't allow it (e.g. if you store your site database, such as a MySQL or SQLite database, on the same server everything that uses that database, you don't need to allow it to the outside world since your site doesn't need to go through the firewall to access a local database).

The third part of the process is to make sure that your server uses secure passwords, and that these passwords are NOT sent across the network in clear text (i.e. make sure they are encrypted when they are being sent across the Internet). There are plenty of posts on this forum for securing your ssh access. Common methods include changing the ssh port if you're allowing passwords to confuse the bots (attackers can easily work around this with a port scan) or to require a private and public key for ssh. You can also combine both methods (change the port AND require ssh keys), though again, a port scan can be used to work around that.

Aside from this, you should check over your system's configuration with a secure system in mind. Even a slight configuration error in anything with network access can potentially (but not always) open you up to attack. It is also good to log into the admin section of any web applications you run (e.g. forums, blogs, CMS's, etc.) and check over the configuration in those to see if there are potential security hazards.

_________________
Kris the Piki Geeker