Linode Forum

[bji]

Hey all. I have noticed some logs on my system that I have never seen before:

Code:
Aug 27 01:49:26 zembla kernel: atack[25830]: segfault at 0 ip
0000000008048e33 sp 00000000ffdb41b0 error 4 in atack[8048000+c0000]
Aug 27 01:49:26 zembla kernel: atack[25836]: segfault at 0 ip 0000000008048e33 sp 00000000ffdb41b0 error 4 in atack[8048000+c0000]
Aug 27 01:49:26 zembla kernel: atack[25835]: segfault at 0 ip 0000000008048e33 sp 00000000ffdb41b0 error 4 in atack[8048000+c0000]
...
Aug 27 01:49:31 zembla kernel: __ratelimit: 53 callbacks suppressed
...
Aug 27 01:49:34 zembla kernel: nf_conntrack: table full, dropping packet.
...


Lots of these "atack: segfault" lines, lots of __ratelimit, and lots of nf_conntrack: table full lines. These all started at 1:49:26 this morning. It all stopped about 23 minutes later with this line:

Code:
Aug 27 02:12:53 zembla kernel: atack[13100]: segfault at 0 ip 0000000008048e33 sp 00000000ffbdf7e0 error 4 in atack[8048000+c0000]


Now as far as I can tell, what the logs seem to be saying is that a process named "atack" ran a ton of times in this 23 minute time period, segfaulted tons of times, and caused lost of network connections that resulted in network connection rate limiting.

This smells alot like someone managed to run something unauthorized on my system. I can't find a binary named 'atack' anywhere on my system so I can only suspect that it was removed already by the attacker somehow.

I searched google for "atack" and find nothing. Does anyone have any clue whatsoever what this is all about?

[bji]

Just some more details, in case they are relevent:

My kernel:

Code:
[root@zembla ~]# uname -a
Linux zembla.ischo.com 2.6.29-x86_64-linode6 #1 SMP Thu Apr 2 15:14:25 EDT 2009 x86_64 Intel(R) Xeon(R) CPU L5520 @ 2.27GHz GenuineIntel GNU/Linux


My services which listen for connections and could be compromised by a remote attacker:

Code:
[root@zembla ~]# netstat -anp | grep LISTEN
tcp        0      0 0.0.0.0:993             0.0.0.0:*               LISTEN      1430/dovecot
tcp        0      0 0.0.0.0:143             0.0.0.0:*               LISTEN      1430/dovecot
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      3452/sshd
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      3545/master
tcp        0      0 :::80                   :::*                    LISTEN      3634/httpd


These are associated with the following versions of software:

dovecot 1.2.1-1
openssh 5.2p1-1
postfix 2.6.2-1
apache 2.2.11-3

Additionally, my apache server has:

php 5.2.10-3
squirrelmail 1.4.19-1
gallery 2.3-1

[Stever]

bji wrote: This smells alot like someone managed to run something unauthorized on my system.
Code: Aug 27 01:46:45 scrith sshd[10582]: Did not receive identification string from 72.14.189.48
Aug 27 01:49:27 scrith sshd[10583]: Invalid user globus from 72.14.189.48
Aug 27 01:49:27 scrith sshd[10586]: Invalid user condor from 72.14.189.48
Aug 27 01:49:27 scrith sshd[10588]: Invalid user tomcat from 72.14.189.48
Aug 27 01:49:27 scrith sshd[10590]: Invalid user global from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10592]: Invalid user upload from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10594]: Invalid user jboss from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10596]: User postmaster from 72.14.189.48 not allowed because not listed in AllowUsers
Aug 27 01:49:28 scrith sshd[10598]: Invalid user demo from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10600]: Invalid user apache from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10604]: Invalid user postgres from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10608]: Invalid user mysql from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10610]: Invalid user tester from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10612]: Invalid user testing from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10616]: Invalid user test from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10618]: Invalid user photo from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10620]: Invalid user oracle from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10622]: Invalid user feedback from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10624]: Invalid user sameer from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10628]: Invalid user temp from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10630]: Invalid user testuser from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10632]: Invalid user portal from 72.14.189.48

Code: $ dig zembla.ischo.com

; <<>> DiG 9.4.3-P3 <<>> zembla.ischo.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41423
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;zembla.ischo.com.              IN      A

;; ANSWER SECTION:
zembla.ischo.com.       86400   IN      A       72.14.189.48

I'd say yes you have been compromised.

[caker]

Also, the kernel you're running has the local root exploit.

http://blog.linode.com/2009/08/17/new-kernels-fixed-vulnerabilities/

You want to select "Latest 2.6 Stable" (i386 or x86_64) in your profile, save, and reboot -- in addition to your other cleaning up.

-Chris

[bji]

caker wrote: Also, the kernel you're running has the local root exploit.

http://blog.linode.com/2009/08/17/new-kernels-fixed-vulnerabilities/

You want to select "Latest 2.6 Stable" (i386 or x86_64) in your profile, save, and reboot -- in addition to your other cleaning up.

-Chris

OK, I did that. Now I am at a complete loss as to how someone got in. I can see from my Linode performance graphs that in that 23 minute range, there was tons of network activity. And then it all goes dead.

Now, there are exactly two users who have a login shell on my server: myself, and root. root's password is quite strong, as is mine. So I don't think that anyone got in directly via ssh.

I'm suspecting that they must have gotten in via a service like apache (most likely), or dovecot or postfix (less likely). In any case, I think they would have been isolated to running with priveleges of the http, postfix, or dovecot user. Unless they took advantage of the local root exploit you mentioned, but if they had, wouldn't I have seen more activity than I did?

I'm just trying to figure out how bad it's likely to be.

Once years ago I got some kind of Apache virus but because it was limited to the apache user, it was relatively easy to clean out. I wonder if I am in the same situation now, or if my entire host is somehow so compromised that I can't trust root anymore ...

[bji]

OK so I've dug around a little, and I just can't see where the exploit could have gotten root access. If it had, I would expect it to remove traces of itself from the logs (if the author was smart), and it didn't. And, I don't see anything else going on with my system that indicates that it was exploited. It's not sending out spam or trying any more ssh attacks.

What I really don't understand is why it stopped after 23 minutes. Why wouldn't it have just kept running indefinitely? Maybe it exhausted its ssh attacks against all other hosts on the local network (sorry everyone!)? And since it didn't have anything else to do without root access, it just quit?

[jsr]

bji wrote:
Once years ago I got some kind of Apache virus but because it was limited to the apache user, it was relatively easy to clean out. I wonder if I am in the same situation now, or if my entire host is somehow so compromised that I can't trust root anymore ...

If you suspect your host is compromised at all, start over with a fresh install. It might seem like a lot of work, but unless you have a reliable way to tell what has been changed (like tripwire) then you are just asking for more trouble. You have no idea what the attacker could have modified and what type of backdoors he could have put in.

By the way, it looks like "atack" is a program to brute force attack ssh on other machines.

http://translate.google.com/translate?hl=en&sl=auto&tl=en&u=http%3A%2F%2Fgathering.tweakers.net%2Fforum%2Flist_messages%2F1355990

It appears to also go by the "brute" name:

http://forum.soft32.com/linux/kernel-brute-segfault-ftopict357104.html

http://nileshbansal.blogspot.com/2006/06/ssh-brute-attack.html

[Xan]

Thanks for the details and advice, bji. It's kind of a frightening story!

[unixfool]

I don't think anyone got into your system (unless you can provide more concrete evidence).

I think what may have happened is that someone attempted a DoS or DDoS against your webserver. The conntrack logs support this. Your system's connection table filled up. If there are tons of connections that the system is attempting to track, the system only has finite resources to track connections...if there are a massive amount of connections and the attack is long-lived, your system will eventually run out of resources and crap itself (dunno if your system locked up or recovered on its own).

Typically, we see this with firewalls attempting to track massive infections such as Slammer or Blaster. While the FW itself isn't infected, the FW will eventually bog down and/or crash. Note that this can happen with any type of connection flood...there doesn't have to be a compromise or intrusion for this to happen.

I'd look at your webserver logs (or your SSH logs, since that port was also showing in your netstat output, although I doubt its an SSH attack that happened).

[bdonlan]

unixfool, the 'atack' logs indicate that they _did_ get into the system, and ran this 'atack' program. Unless there's a legitimate 'atack' program on the system, there really is a problem here.

[unixfool]

If there was actually a system breach, there'd a lot more than what the OP posted.

I conduct first-, second-, and third-level attack and compromise analysis for a living. IMO, kernel logs don't always mean what people think they mean (at least regarding compromise). You can't just run some exploit against a kernel without having a vector (avenue of attack, which is more than likely a network service). Kernel logs don't define the endgame of security, unless it is actually a kernel-level attack, which would suggest that someone is logged in locally to the machine (although this may NOT be the case with the OP). conntrack is a kernel-level function of network connectivity, which means that anyone conducting a DoS will affect connection tracking.

You're doing this admin a disservice if you think that what he described is an actual compromise.

Looking at 'atack segfault' under Google shows the following:

http://www.google.com/search?hl=en&q=atack+segfault&start=0&sa=N

Almost ALL of those links suggest a denial of service and have similar content as the OP posted. If it quacks and waddles like a duck, its either a relative to the duck or IS a duck, IMO.

Here's a GOOD one (from http://kerneltrap.org/mailarchive/linux-netdev/2009/3/23/5223214):

Code: doing a "ping -f -l 3" on my host towards my board on linus tree as of
Friday results in lots of:
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
__ratelimit: 11 callbacks suppressed
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.

for ucc_geth on a MPC832x.
This really looks strange to me, ideas?


If this type of activity is directed against an application, there's where the segfault comes in. Regarding the 'atack' tag, I've no idea where that is coming from but to suggest that someone injected that onto the system is to pull suppositions out of thin air. People in my field actually need data to prove that something like this occurred. That's where application logs

All of this is circumstantial, since there are only kernel logs that were provided. Still...

Secondly, there's the issue of him reinstalling (because he was told to do so) without understanding the nature of the attack. There's a chance it may happen, again, within hours/days of his reinstall if he doesn't understand the who/what/where/why of the prior attack. I've seen machine owners reinstall 3-4 times and get compromised 3-4 times afterward...all because the admins didn't understand the underlying issue.

[jsr]

All signs point to him being compromised though. The atack program is a ssh cracking tool and part of the unixcod package. You can find it at http://www.bart-design.co.uk/unixcod/

There are several other people that have had similar messages appear in their logs and they were all compromised too. I posted a link to one earlier. Here is another that found atack running in "/var/tmp/ ....... / . /unixcod/atack":

http://ibot.rikers.org/%23debian/20080502.html.gz

While I agree he needs to figure out how he got compromised in order to patch and protect against it in the future, telling him that he was simply DoS'd and to not worry about it is not very good advice either.

[Stever]

unixfool wrote: You're doing this admin a disservice if you think that what he described is an actual compromise.
Maybe you missed the strange coincidence that within one second of the OP's segfault log, MY logs (posted earlier) show his system starting an ssh attack against my linode.

Someone got at least user-level access on his system and was looking for more hosts to breach. I don't claim to do this for a living, but if he wasn't compromised then I don't know what the word means.

[bji]

Stever wrote: unixfool wrote: You're doing this admin a disservice if you think that what he described is an actual compromise.
Maybe you missed the strange coincidence that within one second of the OP's segfault log, MY logs (posted earlier) show his system starting an ssh attack against my linode.

Someone got at least user-level access on his system and was looking for more hosts to breach. I don't claim to do this for a living, but if he wasn't compromised then I don't know what the word means.

I agree - unixfool, I think you must have just missed the part of my logs where I listed the lines indicating that a program named "atack" had run locally on my system (and segfaulted quite alot).

What I think happened is that this atack program forked itself tons of times to issue SSH attacks on other hosts on the network simultaneously. My Linode control panel graphs showed that in this 23 minute time period, my system went to 100% CPU usage, tons of network activity, and all of my memory got used up. I suspect that all of the seg faults are due to fork calls failing when the system was out of memory and the atack program not gracefully handling that.

There are three big questions from my perspective:

1. How did they get in? I have searched for known exploits on the handful of services that I am running, and I can't find anything. So maybe there are exploits in PHP or apache or some other software that just isn't known globally yet but is known to some hackers. Or more likely, I just didn't look hard enough when searching for known exploits.

2. Did they acquire any privileges above and beyond the user process that the atack program ran as? Unfortunately, I can't tell what user ran 'atack'. And unfortunately again, I was apparently running a kernel susceptible to the local root explot that was recently fixed in the newer kernels. And yet, I don't see any evidence of having been rooted. It is possible that some process is sitting there on my system completely hidden from me somehow, but I've done pretty exhaustive searches and I don't find anything. netstat shows nothing. process accounting (via acct, which I installed after this happened) shows nothing. All of the find commands and manual file examination shows nothing. The linode control panel shows that my box isn't suffering the CPU, memory, or network load of running any kind of active explot. I basically have no evidence that the user access was elevated to root access by the attacker. I know, the truly paranoid would say that I shouldn't trust anything and should just re-install. And maybe that is a reasonable level of paranoia. But for the time being, I'm watching and waiting to see if there is ever any other evidence.

3. Is my system still vulnerable? Since I don't know how they got in, I don't know if I've "fixed" the problem. I have installed process accounting to help me watch for any further shenanigans ... any other suggestions for stuff to install to help me watch out for a recurrence of this attack?

[justintime]

There's a few things you can do that won't guarantee you weren't rooted, but might help in finding out what happened.

Use the find command to find all files created since that day. If there's too much in that output, try limiting it to all executable files created since that day.

Do you use rpm, apt, or something else? With RPM, you can verify all files against the md5sum from the database to see if any binaries were modified from the packaged version. I'm sure apt has a similar feature.

Find and install chkrootkit.

Find all occurences of '.bash_history', and make sure there's nothing funny in any of those.

Many "hackers" using rootkits will patch netstat, but not lsof. Use it to see if there's any ports listening that shoudn't. If it's already installed, remove it, and reinstall it from a binary package.

Hopefully, by this time you have some files to work with. Whatever you do, don't run them. Copy them off to a different host (like a throwaway VM), and inspect them there. File ownership might point you in the direction of which daemon was exploited.

Hope some of that helps!