I got an alert from linode saying that somebody complained about some phishing code loaded on my site.

Here's the URL (not including my domain) that has the phishing code:

.online.xcgi_F6=1/start.jsplanguage=&first=true.html

Those files had root permissions. I'm looking for ideas on how this happened. I run wordpress 2.5 as www-data. Apache is the only service running.

I'm going to backup my data files, then reinstall the OS on my linode, but in the mean time, I'm just curious how the heck this happened.

I'd recommend that you save the contents of "/var/log" too. While it's possible that whoever broke into your system sanitized that stuff, it's pretty unlikely. Look for programs that crashed.

What version of linux are you running? Was it updated? Have you looked at the list of security advisories for it (especially for services you have running like ssh)?

Were you running plain old wordpress 2.5? Not only does wordpress regularly have security holes, but so many people run it that it's a prime target for hacking.

Vulnerable wordpress plugin? Easy to guess password? Upload scripts?

I've been wading through logs in /var/log all day. No smoking guns yet. But I found thousands of lines in auth.log of people trying to guess my root password.

So I just disabled ssh as root. Am considering switching to public keys rather than passwords for logins.

Perhaps also use on some other port than the default, restrict to certain ip addresses, or as a last resort use something like fail2ban.

It's worth installing fail2ban, or similar. (I use and prefer the simplicity of ssh-guard.)

Disabling password logins is definitely a good deterrent, and recommended. For root logins, you can permit for only networks you use. For example:

# sshd_config
Match Host *.mycompany.com
    PermitRootLogin yes
Match Address 68.198.216.0/24
    PermitRootLogin yes

Your security skill has increased by +1, it will increase again when pubkey auth is enabled and password logins are disabled.

From a recent post:
* Don't use the same passwords everywhere
* Create seperate user accounts for almost everything (ref rule 1)
* Limit those user accounts, strictly.
* Use pubkey auth for all ssh logins, disallow root, (ref rule 3), allow logins only from a small range of IP (try to get a static from your ISP), only allow your user to login via ssh from your IP address.
* Limit your external servers, learn /etc/hosts.allow and /etc/hosts.deny
* Learn IPtables, or setup an easy bash interface.
* Install one of the hundred brute-force/dictionary attack blockers (fail2ban, denyhosts, etc)
* Seperate your mysql users and only give them rights enough on their own databases to do just they things they need to do.
* Use good and well tested software, bleeding edge isn't always the best choice.

This is just generic basics, there is plenty more.

A reinstall is the only way to know for sure your host has been cleaned. A hassle, but worth the peace of mind.

I've been seeing a few log attempts with /wp-comments-post.php resulting in a 500 error, which concerns me -- and I'm running the latest wordpress.

You don't need the complexity of a fail2ban, I prefer using a few lines of iptables. Stops brute force attacks in their tracks. http://www.ducea.com/2006/06/28/using-iptables-to-block-brute-force-attacks/

Unfortunately ip6tables doesn't seem to have recent match support yet. But if you're not running IPv6, no problem.

I love this forum. Always loaded with good ideas.

btw, I've recently started to use OSSEC -- couldn't be easier to install and set up and in its default config, quite effective. Even blocks too many failed SMTP requests. Looks for loads of stuff and emails you alerts when fishy stuff is found.

Has a rootkit detector as well, although if someone gets in as root and quickly kills the monitoring process, you're out of luck, along with a bunch of other troubles.

"Nuke it from orbit, it's the only way to be sure."

For SSH logins, in my opinion, public key is the only way to fly.

_________________
Bus error (passengers dumped)

I agree that reinstalls should happen after a system compromise, but only AFTER it is understood how the machine got compromised in the first place. I've seen several people reinstall only to get immediately compromised again. Sometimes its an app that they repeatedly install that's the culprit (buggy code). Other times it is something the admin did that repeatedly caused the hole (buggy brain).

Reinstalls aren't the solution. It is only the way to start from scratch after a root cause has been discovered, though.

unixfool is right, and linodes help out tremendously in case of a compromise. One could log into lish and shut off networking on an infected linode to contain damage, then buy a new linode temporarily and reinstall it to get your services back online asap. Then when that is done, time can be spent analyzing the isolated, infected linode to see what went wrong. When analysis is complete, that linode can be destroyed leaving you with the new linode in its place.

If your new linode is in same datacenter I'm sure the staff could probably set up DHCP with the same IP as your old linode and remove the public IP from the infected linode so DNS cache times won't be an issue.

You can swap IPs between Linodes using the utility on https://www.linode.com/members/linode/network.cfm