Recently, the webserver on my linode was hacked to send 70,000 ebay 'fishing' emails.
I became aware of this in a very short time as the bounces for invalid addresses on the list the attacker used went back to my www-data, which is aliased to root, which is aliased to my account. Thus, I got about 25,000 bounce emails in my personal account, which caused pine to die horribly by memory exaustion when it tried to open my inbox.

Since then, I've added many new security measures to my webserver designed to stop this sort of thing from happening again.
1) PHP reconfigured (php4)
* I've used safe_mode to disable mail(), and exec() in php <--- though I might turn this back off in a few days, as it breaks a lot of functionality in user websites.
* I've set harsh process limits (e.g. max runtime for scripts = 30s) on php
2) Httpd reconfigured (apache)
* I've disallowed SSI on user webpages (I did have IncludesNoExec)
* I've set apache to log ALL access to the webserver
3) MTA reconfigured (exim)
* I've reconfigured my MTA not to count the webserver as 'trusted'.... so all outgoing emails sent via the webserver have "myhost WEBSERVER" tagged as their "From".... thus making fishing scams pointless as they would be easily detected.

I'm currently using debian linux. I'm shortly moving to an OpenBSD dedicated server, where additionally, I will have :
1) a fully chrooted apache
2) mod_security loaded
3) process accounting
4) fewer shell users

Does anyone have any other tips for how I can prevent people taking advantage of my webserver like this?
The provisions :
1) I still have to give relatively untrusted people shell access. Unfortunately, that's the nature of my business.
2) I still have to allow them to have websites with userdir, php, and mysql.
3) I should, if at all possible, still allow the sending of mail via www-data.

I've tried looking in my apache's logs...... I can't find anything out of the ordinary, and I've tried looking in my MTA logs, it just identifies all the mail as coming from www-data.

What I'd really like is some kind of script so that any mail sent via www-data is tagged with the location of the file or program which caused that mail to be sent, so for example, I could look at any future fishing emails and see that they were tagged as :
SENT : via www-data : mail() invoked at /home/naughtyuser/public_html/muahaha.php like 24

Anyone know if this is possible? I'm really angry at this guy for spamming through my webserver. I've not been able to track him down, even resorting to rather extreme measures like searching for his scripts via grep -ri 'ebay' /home/* .

Any suggestions for how to make such spamming from my webserver more difficult or impossible in future, especially those that mean I would end up with reliable logs as to who is doing it, would be much appreciated.

You may need to enable PHP4 as CGI rather than an Apache module. And then you can use the suexec feature of Apache to make all scripts run under the userid and groupid of the actual user of each vhost.

Also, check out the libapache2-mod-suphp package. I must confess I haven't heard of this before -- only came across this as I did an apt-cache search just now.

Wow! Sounds like a horrible situation. Make sure to go to check your IP at Spamhaus, rbls.org, and other blacklists. I doubt a mere 70k e-mails would get you on there, but you never know.

As far as a solution, I can't think of a good one! I think you'll need to do it on the MTA level, exim in your case, because I don't think blocking mail() blocks things like the Mail PEAR package (which can use SMTP directly).

Do you allow perl or shell scripts to run as CGI scripts? If so, they could always use those as well.

Send-rate limiting seems like the best bet to me, but as a Postfix user, I'm not sure how to accomplish that in Exim.

And one last note: I hope you can find and sue the bastard.

I'm soon to move to a fully dedicated OpenBSD server which has chroot and suexec enabled. For the moment, I will look into libapache2-mod-suphp, and configuring exim to block or limit mail from www-data.

I'm already blacklisted on spamcop and enough people have complained that I'm getting abuse@ and postmaster@ emails about it. Thankfully I dealt with days earlier then the people emailed me about it. I really don't want to be in the position of the guy who signs on one day to find his server offline because someone exploited it in his absence and he wasn't online to deal with it right away.

Grrr, I still haven't figured out how he caused the mails to be sent other then either :
1) a shell user who put a malitious script in their webdir and ran it via apache
or
2) someone who exploited the httpd.

I'm turning on logging down to the level of info and debug now...... but I'm still mad, and probably won't feel fully satisfied until I catch the git responsible.

I contacted the hosting provider that was hosting the images the scammer was using, and they terminated the guy's hosting.

From now on, www-data won't be trusted, or allowed to send mail.
I've already got suexec running for perl, now all I need is to get suexec running for php, and I'm sorted Oh, and www-data won't be trusted to send mail anymore.


------------------------------------------------------

Abuse@interland.com to me Apr 7 (4 days ago)
Hello Ash,

re: http://www.cc4ce.net/images/

Please email abuse@interland.com for any type of fraudulent activity from an IP in our lease.

I disabled those phishing scam images for you and I appreciate you letting me know. I am trying to contact the owner of the site to let them know of this compromise.

Regards,
Marnie
Interland Policy Enforcement Group
Interland, Inc.
Abuse@Interland.com

At least you're not using Sendmail...

If security is priority #1, consider switching to qmail (netqmail 1.05). Postfix is a decent 2nd choice and Exim would be my third choice--again, this is based on security being priority #1 above all else. The number of expoits found in each of these speak for themselves as one quantifiable measure of security.

Also, try using the latest mod_security (not the outdated buggy one in Debian packages). And use a custom ruleset--not just one of the default ones with minimal rules. You should include some of the more popular snort rules translated to mod_security.

Run the hardening script called Bastille Linux. It will help you improve the security of your linux distro (there's a debian package for it).

Run a firewall like Shorewall or Firehol. If your server has no business initiating outgoing connections to unknown servers, then why allow it? This is in addition to dropping all incoming connections to unused ports.

Completely uninstall compilers and scripting languages you don't need (like gcc). At a minimum, change permissions so only root can use them.

You may want to have something else answer web requests BEFORE apache. I prefer pound 1.8 reverse proxy. Very useful, doesn't read/write to disk directly except for configs during startup and is easy to chroot.

Which version of Debian are you using? Stable? Testing? Unstable? Strangely enough, testing (sarge) might be the worst choice for security because of the delay in getting fixes compared to both stable and unstable--but that is probably going to change this year.

You may want to consider CentOS 4 (free RHEL 4 clone). I'm a Debian user that disliked RHEL 3 but I'm seriously looking at RHEL4 & CentOS4 because of the new security features taken from SE Linux.

I really suggest you get suphp.
That will give you a lot of useful log info in your MTA log.

I really want suphp, but I can't find a way to get it working on my new (non-linode) server. (I'm moving up from VDS), partly as linode doesn't support the OpenBSD OS I want. Ironically, all the setup help for it I can find is for debian, the OS I'm moving away from!

I use postfix (I avoided qmail due to what I heard about the bad state of its OpenBSD ports)..... and the latest mod_security, and Bastille doesn't work on BSD.

I have a firewall written in iptables atm, and will write a new one in OBSD's PF as soon as I learn it. Compilers are restricted to root on both systems.
I'm running debian stable atm, but will soon run OBSD 3.6.

I've got a lot of other OSes on my desk, and I'll add CentOS to my list of ones to try

Thanks for all the suggestions

http://www.suphp.org

Download, compile and install manually.
I got it working on my home test server that has FreeBSD. (Sorry, I don't know about Open BSD )

Considering this sounds like a serious config problem, switching to a secure *BSD wont help.

Security wont mean diddly if services aren't setup and secured correctly.

The problem was that several things interacted :
1) php's mail() function was enabled
2) php scripts ran as the apache user
3) there was no rate limiting on outgoing mail
4) the www-data user was allowed to specify any 'from' address for outgoing mail.

All these are default settings, and this is the first time they have ever been abused like this. Needless to say, I now have suphp running on my new server as well as suexec, my apache is chrooted, and any mail from www-data is identified as such. I even went as far as disabling php's mail().

The reason I'm switching to a secure *BSD is that when security is your top priority, it makes sense to use an OS for which security is also the #1 priority, rather then trying to adapt an OS with a different focus to security. I don't think using a more secure OS will magically solve all my problems, but it will help... by, for example, being easier to secure and having better embedded security support.