My CPU usage jumped from an average of 4% to 104% yesterday, and has been sitting around 104% ever since.

I've got one rails app running on there (ubuntu, nginx, passenger), which has really low traffic (this hasn't changed - about a couple of hundred requests/day).

I haven't touched anything on the server in the past week either.

Has anyone experienced anything similar?

Well, does ''htop', 'top' or 'ps' say anything interesting?

104% is a very suspicious number. I bet the 100% is from something single-threaded chewing as much CPU as it can, and the other 4% is everything else.

_________________
Matt Nordhoff (aka Peng on IRC)

Thanks mnordhoff - Yep - I have discovered that the 100% was coming from a single perl process. How it got there I have no idea. I also discovered a whole lot more. Lots of request for phpmyadmin and other setup scripts in the log files, and requests for odd domains.

Also a persistent IRCD connection from a atw.hu domain.

In short - the server has been compromised. Backed everything up, and about to rebuild.

Before I delete everything - I'd like to know how they got in though. Do you know where I could find the tell tale signs?

Well that's normal. Any public IP suffers a lot of attacks; the important part is whether or not they succeed.


OK, that's definitely not normal! Yikes.

(You're sure it's really compromised, not just an attacker attempting to connect or something?)


Sorry, that's not something I know much about.

_________________
Matt Nordhoff (aka Peng on IRC)

1. Back up the entire node (linode's backup service would be ideal for this)
2. Clone the node from the backup
3. Boot the node using finnix so it can't talk to the outside world
4. Check your log files

That's a good start.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue