Hello,
All of my sites are showing Database error message. Upon trying to access phpMyAdmin it opens a survey page (redirect) & not able to access server via SSH/putty. I shut down the server as to absorb what has been done & what to do now.
The linode support dont offer anything much apart from throwing some links of library.
I think I dont have time nor have confidence to recover and deploy the hacked system.
I'm ready to shell extra money for Linode managed but they do not offer a recovery either.
How would you recover your linode system if it's unaccessible via ssh. (I could login via lish? but I shut the server)

Delete the node and recover from known safe backups. It's the only way to be sure.

But how to recover the data?
shouuld I download the linux via SFTP?

You don't - you get the last known good (and non-compromised) backup and use that.

_________________
Either provide enough details for people to help, or sit back and listen to the crickets chirp.
Security thru obscurity is a myth - and really really annoying.

First rule of compromise recovery: no bits leave the compromised node. You don't export data, you don't hook it up to a network, you don't start services on it. If anything, you attach it to a clean environment (like rescue mode) and investigate the image(s) to glean knowledge. But you're doing that to try to learn what went wrong, not to actually move any bits off the compromised system onto good systems.

- Les

"I say we take off and nuke the entire site from orbit. It's the only way to be sure."

My user name password is changed. I changed the root password & rebooted linode in rescue mode & now downloading the entire image.
I think it would 've been easier if I had static html sites but I've around 10 wordpress sites, restoring these would be a problem since I dont have backup system in place.

All right, I've change password for my user. Should I dare to boot my linode now?
How to check which part is corrupted? how to see the log files?
all my sites are showing database error. Upon trying to visit phpmyadmin url it redirects to some survey.

Absolutely not.

As others have said, your machine is compromised, and you have no way of knowing which bits have been altered. Re-install from the ground up and restore from backup.

For more info on how to appropriately respond, check out this Q/A over on Serverfault on how to deal with a compromised server.

_________________

Then you my friend are totally f**ked.

Why in the world would you run 10 wordpress sites and NOT have a backup (both files and databases)?

And why are you downloading the hacked image? Unless you plan on running forensic diagnostics to determine how you got screwed, it's completely useless.

_________________
Either provide enough details for people to help, or sit back and listen to the crickets chirp.
Security thru obscurity is a myth - and really really annoying.

Good article except for the part that says "Don't Panic".

Compromised with no backups seems to be the perfect time to panic.

_________________
Either provide enough details for people to help, or sit back and listen to the crickets chirp.
Security thru obscurity is a myth - and really really annoying.

Johnycage,

About 5 different people, including Linode staff, have suggested deleting this image and restoring from a backup. We are not suggesting this because we are assholes and want you to lose your work. We are suggesting this because that machine has been cracked and it's full of malware, botkits, and all other kind of nasties. Getting anything off it is a very risky job. You can try if you want but you should know that the most experienced amongst us would exercise extreme care if we did it at all, which in all likelihood we would not. Whatever you do -DO NOT BOOT THE IMAGE-.

In future you need backups. Even if it's just shutting things down periodically and using the Linode manager to copy the disk images it's a lot better than nothing. Really backups should be off-site. There are lots of ways to do it. I like BackupPC personally.

You need to do security updates for all your software.

You need some kind of monitoring, in the crudest case that could be nosing though the system periodically to look for changes you didn't make or processes that should not be running. Tripwire is good but it's a PITA to setup.

When people come to me asking me to recover their system, I tell them to create a new node and restore a backup to that, also I keep the existing node so I can find the cause of the vulnerability, there's no point in restoring a backup and not patching whatever hole caused the problem in the first place.

If they don't have backups then things get a whole lot more complicated, in the case of wordpress it's best to create a new node, reinstall wordpress, reinstall any plugins/themes, then if you have any custom code go through it line by line looking for anything weird. You also can't trust any images uploaded you'll need to reprocess those to make sure they don't contain anything weird. Lastly you restore your database after checking each and every entry for anything weird. It's a pain staking process.

In the future have backups, at a minimum the Linode backup service will do it's not expensive. I personally use duplicity.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

Your server has been corrupted by Chaos. The Ordo Scharzenkommando recommends Exterminatus.



I'm totally serious though. Do not boot the disk image. It's very easy for a skilled hacker to hide malware. No one is going to leave backdoor.py in your home directory.

Also, always make backups, daily if possible. Run a script every day to back up databases and put them in some database folder. Use rsync on a local computer to back up all the files and databases. Use a local backup system on your own computer that stores the versions of the files for each day so that you get a history of backups. Getting a daily backup of the server can be made really simple; it's very important to do. At least pay for Linode backups.

I'm planning to write about security advice here on the website for my GitHub profile:
http://inquisitor-sasha.sturmkrieg.ru/

Here's the source for it if anyone wants to contribute; I can add push access
https://github.com/Inquisitor-Sasha/inq ... .github.io

And for backup advice and GitHub, upload any custom software that you use to GitHub so that you don't have to recreate it if you lose the linode. It will be open source, but unless you're actually planning to sell it, you might as well make it free for other people to use.

EDIT

Do not use PHPmyAdmin. It is a total backdoor.

_________________
Homepage www.sturmkrieg.com
Social network Gamernet
Development website Sashaweb Development
Imageboard img.sturmkrieg.com
WikiHub free wiki host Community Wiki