Hi all

Since January I've been kept up nights by the infection of my server by RATs (Remote Access Trojans).

Since my first infestation, which manifested itself in random PHP files appearing in the root of one of my websites which are hosted on this machine and SPAM emails sent from the mailserver (I'm running ISPConfig on Debian Squeeze with Postfix and Apache), I have spent hours trying to solve the problem.

My latest action was to take out a second linode, re-install ISPconfig, then migrate sites over. Only one of my websites was getting infected (although they all run on the same application which is called ModX - a CMS). This was the site I migrated to the newly built linode.

When I migrated the site, I took a backup of the SQL database and site files, and built a clean installation of ModX onto the virtual host and imported the CMS database.

Sure enough within a week, I start receiving Linode notification emails for high use of bandwidth. This morning my bandwidth and IO graphs were maxed-out for about 3 hours! There are rogue files in the web root, and my mailq has been absolutely hammered by SPAM attempts!

Whatsmore on my new linode as of today, ISPConfig web interface is broken, and the site-specific FTP account I set up no longer works! I can still SFTP to the server using root, so I'm reasonably confident the damage has only been done to the virtual host in question rather than the whole OSE being compromised.

As part of the migration, I set up an entirely new set of passwords, not a single password was used again. i.e. passwords for root, FTP accounts, mysql user and su accounts, etc.

Where can I go for help with this kind of thing? I feel I've read so much about RATs online, and everyone just says I need to ensure passwords are changed, systems are kept up-to-date, only use secure FTP, etc. - all of which I'm following religiously. Needless to say I've been watching every security update for ModX and the very latest version was installed on my new linode a few days back.

Anyone else have any experience with this?

I'm guessign there's your problem:


so you created a clean setup, and then copied the infected website over?

deleted

I'm not sure what you have is RATs unless they have enough access to actually administer your server.

Find out how these files are getting in your web root and stop that happening. Read your web logs for clues. Do you allow users to update files of any kind or allow them to enter data that could be interpreted as PHP when it's displayed later?


Post the URL here and someone here will have a look at it.

PHP Shells and stuffs can be made pretty much useless if you disable these functions. With PHP-FPM:

php_admin_value[disable_functions] = apache_child_terminate, apache_setenv, define_syslog_variables, escapeshellarg, escapeshellcmd, eval, exec, fp, fput, ftp_connect, ftp_exec, ftp_get, ftp_login, ftp_nb_fput, ftp_put, ftp_raw, ftp_rawlist, highlight_file, ini_alter, ini_get_all, ini_restore, inject_code, mysql_pconnect, openlog, passthru, php_uname, phpAds_remoteInfo, phpAds_XmlRpc, phpAds_xmlrpcDecode, phpAds_xmlrpcEncode, popen, posix_getpwuid, posix_kill, posix_mkfifo, posix_setpgid, posix_setsid, posix_setuid, posix_setuid, posix_uname, proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, shell_exec, syslog, system, xmlrpc_entity_decode, mail

Note that the last value here is mail, this will prevent the mass mailing but it is a bit tricky if you also send out legitimate mail through PHP. You may be able to reconfigure your application to send mails through an external mail server instead. Either way, the above should help, even if you exclude the mail function

GL

Or something like this:

http://www.cyberciti.biz/faq/linux-inot ... rectories/

and cp all relevant logs to a safe location, dump netstats and whatever else...

I hope/presume that you keep the online applications up-to-date, in this case ModX and whatever eles you maybe running. Open source applications have the problem that if you do not patch them amateur hackers can gain acces through known security holes.

_________________
Linode 512 @ cbr250.com

I don't think there's my problem. I may be wrong, but the infections in question took the form of some base 64 encoded php files in various directories under the web root of the given vhost.

When I inspected these naughty php files (FTP to my laptop), they were flagged up in my virus scanner (Avast for Win 7). I ran the sql backup file through the same virus scanner and it looked clean. Also I checked it for any base 64 encoded crap and there wasn't any.

But yes - I've no doubt it would be better to re-author all my sites from scratch. Something that would take weeks...

Thanks for the helpful words, all.

Here's what I did:

- installed incrontab, this is awesome as it detects file changes in a given directory then runs a script accordingly which emails me about any changes
- I keep postfix stopped, then login periodically to check the mailq is clean then start postfix when I'm happy, then stop it again (not ideal, right?)
- When I notice huge traffic spikes, I try to log in and run iftop -n to see where connections are coming from and how much bandwith they use, then ban them using e.g. iptables -A INPUT -s 65.23.85.110 -j DROP
- I installed and ran phpsecinfo to help secure my php installation

The above has basically made the problem go away for the last 4 weeks now. But it's not ideal, right?

I've looked through apache logs and cross-referenced them with the datetime of the infected file to see what IP address it came from. I can check this is when the huge traffic spikes start.

But my expertise stops there.

Can someone help me to determine exactly how the files got there? Through what process? What insecurity?

Any help greatly appreciated.

mrmat,
It's almost impossible to determine what happened at this point, even if we had access to the server and all of the logs. I assume you've changed all the passwords for accessing the site, including root, FTP, email users, etc.

If all of your sites are running of the same CMS and only one is getting infected it might help you find the issue by focusing on the differences between the sites (file & directory permissions ownership, CMS settings, software versions, etc).

Have you searched for security issues and/or other ModX sites being infected in this way?

MSJ

What version of modx are you using?
There was an execute arbitrary code thing until version 0.9.6.2.

http://www.cvedetails.com/cve/CVE-2008-5938/

Thanks all

I'm running version 1.0.9 (from only a month ago, I think there's a new version now, I'll update), either way it's well past v0.9.6.2

@Main Street James, yep - I've changed every single credential on the server and replaced them with ridiculously strong passwords. Also I have standardised on a file/dir permissioning model across all sites and researched permissions quite heavily, not to mention standardising on CMS settings etc.

I think I will go through once again with a fine toothcomb all the scripts, modules, plugins, etc. to check their versions.

Thanks, I'll keep battling. Problem is, whenever there is a compromise, I have to set aside more than a few hours to rebuild a second server from scratch then migrate sites one by one - argh!

mrmat,
If you are still getting compromised on that one site:
- set up a brand new site.
~~ use a subdomain or even a throwaway domain (still cheap based on the amount of time you're losing).
- clean up the infected files on the problem site.
- port only the data from the problem site to the new site.
- sit and wait for the problem site (or both sites) to get compromised.

If you only bring over the data from the problem site and the new site gets compromised then you know it is something in the data. If the new site doesn't get compromised then you copy the current data from the problem site over to the new site and move/rename the new site's code to replace the problem site (be sure to have a backup of the problem site just in case). You may have to chown the files so the newly copied files match their new location/owner.

If the problem is hidden in a post, comment, etc of a specific page of the infected site then the new site would eventually end up compromised when the remote system requests that page/post (it wouldn't happen under the test domain/subdomain because it wouldn't get requested on that unknown domain/subdomain).

Have you checked the logs to see what page on the problem domain was accessed around the same time as the timestamp of the infected files that end up on your site?

BTW, have you checked the .htaccess file of the problem site? The shenanigans could be hidden in there. Have you checked all your cron jobs? Are there any you don't recognize? If the infection happens very regularly or predictably it may be a cron job.


MSJ

@MSJ

Really great post; some very sensible ideas I hadn't thought of.

One question, when you say "only bring over the data", do you mean, copy the files but not the website database?

That strikes me as a really good idea, one I hadn't thought of.

Thanks again,

Mat

Only bring over the database, but don't bring over the executable code. (This assumes you've inspected the contents of the database for, say, unauthorized users and the like.) Re-deploy the executable code from trusted sources.

_________________

/* TODO: need to add signature to posts */