OK, so I was able to find some more out.

My machine does nightly backups and I compared a nightly backup from before the break-in to one afterwards. I found some "interesting" files in /tmp:

-rwxr-xr-x 3 88 88 464140 2007-09-20 10:04 k-rad3
-rwxr-xr-x 3 88 88 8550 2008-02-18 05:33 2.6.20
-rwxr-xr-x 3 88 88 11048 2008-02-18 05:34 2.6.23
-rwxr-xr-x 3 88 88 11523 2008-02-18 05:36 2.6.24
-rwxr-xr-x 3 88 88 10137 2008-02-18 05:37 2.6.21
-rwxr-xr-x 3 88 88 240642 2008-02-18 05:48 2.2.4
drwxr-xr-x 3 88 88 4096 2009-08-25 02:15
drwx------ 5 88 88 4096 2009-08-25 07:00 -

Also some "hidden" files:

./ /.time:
total 636
-rw-r--r-- 1 88 88 328 2009-08-27 03:00 72.14.189.48.user
-rw-r--r-- 3 88 88 328 2009-08-25 02:12 72.14.189.48.user2
-rw-r--r-- 1 88 88 328 2009-08-27 03:00 72.14.189.48.user3
-rwxr-x--x 3 88 88 317 2006-10-29 22:15 autorun
-rwxr-x--x 3 88 88 492135 2006-10-29 22:15 bash
-rw-r--r-- 3 88 88 47 2009-08-25 02:11 cron.d
-rwxr-x--x 3 88 88 9175 2009-08-25 02:11 inst
-rw-r--r-- 3 88 88 171 2009-08-25 02:13 LinkEvents
-rw-r--r-- 3 88 88 14 2009-08-25 02:11 mech.dir
-rwxr-x--x 3 88 88 22882 2006-10-29 22:15 m.help
-rw-r--r-- 1 88 88 1043 2009-08-27 03:00 m.lev
-rw------- 3 88 88 6 2009-08-25 02:11 m.pid
-rw-r--r-- 1 88 88 2127 2009-08-27 03:00 m.ses
-rw-r--r-- 3 88 88 2783 2009-08-25 02:11 m.set
-rw-r--r-- 1 88 88 7699 2009-08-27 03:30 ning.seen
-rw-r--r-- 1 88 88 11738 2009-08-27 03:30 niu.seen
drwxr-x--x 2 88 88 4096 2007-05-23 20:00 r
-rwxr-x--x 3 88 88 29 2006-10-29 22:15 run
-rwxr-x--x 3 88 88 752 2008-06-01 12:06 start
-rwxr--r-- 3 88 88 169 2009-08-25 02:11 update
-rw-r--r-- 3 88 88 13 2009-08-25 02:11 vhosts
-rwxr-x--x 3 88 88 28489 2006-10-29 22:15 xh
-rw-r--r-- 3 88 88 0 2009-08-25 02:12 zhou.seen

On the host that has the backups, user 88 and group 88 does not exist. But on my Linode, from which the backup was made, 88 is:

postgres

So it looks like they exploited some way to get into postgres.

One thing I discovered today is that I had some users that didn't allow interactive ssh (I set the shell for all users to /bin/false in /etc/passwd, except for me and root) but I didn't realize that they could still be used to create SSH tunnels. So it's possible that the attacker was able to use one of these accounts to SSH tunnel into my local postgres which only listens on 127.0.0.1 and has an easy password.

I have now added an AllowUsers line to /etc/ssh/sshd_config to disallow SSH access from anyone except me and root.

Another interesting fact:

It looks like the earliest file created was on
2009-08-25 07:00. So I guess that this exploit had been around for a few days before it ended up being "used" to run an SSH attack.

Also - after a reboot, these files were cleaned out and no longer exist on my Linode.

I believe that the attacker did not get root access; if they did, they probably would have removed all evidence of their attack from /tmp.

Do you have the postgres running and its port (5432) open to the world? If so, you should setup some firewall rules to block it off.

By the way, I know next to nothing about postgres, but some databases (MS SQL) provide methods to allow you to execute a system command from within a query. If you have a weak admin password and postegres is open to the world, then you are just asking for trouble.

It is running but it is only listening on 127.0.0.1, so it cannot be accessed from outside.

It does have a weak password, but I assumed that because it was only listening locally, it could not be exploited from afar.

However, if someone was able to set up an SSH tunnel from the local postgres port to some remote host, because they were able to guess the password for one of my email users who shouldn't have been allowed to SSH at all (I've fixed this), then they could get to it.

I will change my postgres password to be more secure in any case.

I didn't know that, even after googling for 'atack'. Thanks!



In no way did I say that it wasn't anything to worry about. I specificially stated that with the information that was provided, the issue was difficult to analyze. Up to the point where I chimed in, there was nothing in the thread that suggested what you did up above. I live in a world where what's in your face is what you act upon...no data to act on hinders the process of analysis. Based on the facts provided at that time, it appeared to be a DoS to me, especially looking at the conntrack log entry.

As I said before, just posting an obscure kernel log entry by itself doesn't mean that a machine has been cracked.

Thanks for updating this thread with further info, though...it helps a ton in understanding what's going on.[/quote]

You didn't exactly say what those logs were, either (although I'd wondered WTF they were and what they had to do with the issue). I'd assumed that you were seeing something similar to what he was...I didn't factor in that he was your network neighbor, but then again, why should I if you didn't specifically mention it?

Thanks, though.

This is AWESOME information!

Looks like they loaded and installed a kernel-level local exploit (http://www.xfocus.net/tools/200512/k-rad3.c). It looks like they loaded up several exploits that targets different kernel versions. I also see what looks like an IRC bot (mech.dir hints to this and http://www.webhostingtalk.com/showthread.php?p=5323420 shows an example). The *.seen files look to be bot log files of bot users.

Man, your machine was jacked!

I don't relish carrying this particular thread of discussion any further but ... he included "dig" output next to his logs to show that the host that was issuing the SSH attacks was my host (zembla.ischo.com). So all that you needed to know to make the correlation was in his posts.

That being said, I paid alot closer attention to his post than you probably did because I was the one who was compromised and so I had a more than intense interest in the thread. I can see how, if you were only reading casually, you might have missed it.

Yes, I examined all of the files that were present and you are right, the k-rad and kernel version files were executables which attempt to gain root access via various outdated Linux kernel bugs. None of them would have succeeded against my kernel, though.

Apparently I was running a kernel that is susceptible to the newest Linux kernel exploit, but they didn't run that exploit. I guess their scripts were a little too old.

I have compared SHA-1 hashes of all files that existed on my system before and after the compromise. There were no modified files aside from those that I listed above from /tmp. That, combined with the fact that there is no other evidence whatsoever of continued abuse of my system, leads me to feel fairly confident that the attack has been neutralized simply by rebooting into the latest Linux kernel, removing all files from /tmp (which the reboot accomplished), and putting more stringent controls for sshd (via AllowUsers).