Linode Forum
Linode Community Forums
 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Post new topic  Reply to topic
Author Message
PostPosted: Thu Aug 27, 2009 1:34 pm 
Offline
Senior Member

Joined: Thu Aug 28, 2003 12:57 am
Posts: 273
Hey all. I have noticed some logs on my system that I have never seen before:

Code:
Aug 27 01:49:26 zembla kernel: atack[25830]: segfault at 0 ip
0000000008048e33 sp 00000000ffdb41b0 error 4 in atack[8048000+c0000]
Aug 27 01:49:26 zembla kernel: atack[25836]: segfault at 0 ip 0000000008048e33 sp 00000000ffdb41b0 error 4 in atack[8048000+c0000]
Aug 27 01:49:26 zembla kernel: atack[25835]: segfault at 0 ip 0000000008048e33 sp 00000000ffdb41b0 error 4 in atack[8048000+c0000]
...
Aug 27 01:49:31 zembla kernel: __ratelimit: 53 callbacks suppressed
...
Aug 27 01:49:34 zembla kernel: nf_conntrack: table full, dropping packet.
...


Lots of these "atack: segfault" lines, lots of __ratelimit, and lots of nf_conntrack: table full lines. These all started at 1:49:26 this morning. It all stopped about 23 minutes later with this line:

Code:
Aug 27 02:12:53 zembla kernel: atack[13100]: segfault at 0 ip 0000000008048e33 sp 00000000ffbdf7e0 error 4 in atack[8048000+c0000]


Now as far as I can tell, what the logs seem to be saying is that a process named "atack" ran a ton of times in this 23 minute time period, segfaulted tons of times, and caused lost of network connections that resulted in network connection rate limiting.

This smells alot like someone managed to run something unauthorized on my system. I can't find a binary named 'atack' anywhere on my system so I can only suspect that it was removed already by the attacker somehow.

I searched google for "atack" and find nothing. Does anyone have any clue whatsoever what this is all about?


Top
   
 Post subject:
PostPosted: Thu Aug 27, 2009 2:00 pm 
Offline
Senior Member

Joined: Thu Aug 28, 2003 12:57 am
Posts: 273
Just some more details, in case they are relevent:

My kernel:

Code:
[root@zembla ~]# uname -a
Linux zembla.ischo.com 2.6.29-x86_64-linode6 #1 SMP Thu Apr 2 15:14:25 EDT 2009 x86_64 Intel(R) Xeon(R) CPU L5520 @ 2.27GHz GenuineIntel GNU/Linux


My services which listen for connections and could be compromised by a remote attacker:

Code:
[root@zembla ~]# netstat -anp | grep LISTEN
tcp        0      0 0.0.0.0:993             0.0.0.0:*               LISTEN      1430/dovecot
tcp        0      0 0.0.0.0:143             0.0.0.0:*               LISTEN      1430/dovecot
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      3452/sshd
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      3545/master
tcp        0      0 :::80                   :::*                    LISTEN      3634/httpd


These are associated with the following versions of software:

dovecot 1.2.1-1
openssh 5.2p1-1
postfix 2.6.2-1
apache 2.2.11-3

Additionally, my apache server has:

php 5.2.10-3
squirrelmail 1.4.19-1
gallery 2.3-1


Top
   
PostPosted: Thu Aug 27, 2009 3:08 pm 
Offline
Senior Member

Joined: Fri Dec 07, 2007 1:37 am
Posts: 385
Location: NC, USA
bji wrote:
This smells alot like someone managed to run something unauthorized on my system.

Code:
Aug 27 01:46:45 scrith sshd[10582]: Did not receive identification string from 72.14.189.48
Aug 27 01:49:27 scrith sshd[10583]: Invalid user globus from 72.14.189.48
Aug 27 01:49:27 scrith sshd[10586]: Invalid user condor from 72.14.189.48
Aug 27 01:49:27 scrith sshd[10588]: Invalid user tomcat from 72.14.189.48
Aug 27 01:49:27 scrith sshd[10590]: Invalid user global from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10592]: Invalid user upload from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10594]: Invalid user jboss from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10596]: User postmaster from 72.14.189.48 not allowed because not listed in AllowUsers
Aug 27 01:49:28 scrith sshd[10598]: Invalid user demo from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10600]: Invalid user apache from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10604]: Invalid user postgres from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10608]: Invalid user mysql from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10610]: Invalid user tester from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10612]: Invalid user testing from 72.14.189.48
Aug 27 01:49:28 scrith sshd[10616]: Invalid user test from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10618]: Invalid user photo from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10620]: Invalid user oracle from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10622]: Invalid user feedback from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10624]: Invalid user sameer from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10628]: Invalid user temp from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10630]: Invalid user testuser from 72.14.189.48
Aug 27 01:49:29 scrith sshd[10632]: Invalid user portal from 72.14.189.48

Code:
$ dig zembla.ischo.com

; <<>> DiG 9.4.3-P3 <<>> zembla.ischo.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41423
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;zembla.ischo.com.              IN      A

;; ANSWER SECTION:
zembla.ischo.com.       86400   IN      A       72.14.189.48

I'd say yes you have been compromised.


Top
   
 Post subject:
PostPosted: Thu Aug 27, 2009 3:24 pm 
Online
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
Also, the kernel you're running has the local root exploit.

http://blog.linode.com/2009/08/17/new-k ... abilities/

You want to select "Latest 2.6 Stable" (i386 or x86_64) in your profile, save, and reboot -- in addition to your other cleaning up.

-Chris


Top
   
 Post subject:
PostPosted: Thu Aug 27, 2009 4:27 pm 
Offline
Senior Member

Joined: Thu Aug 28, 2003 12:57 am
Posts: 273
caker wrote:
Also, the kernel you're running has the local root exploit.

http://blog.linode.com/2009/08/17/new-k ... abilities/

You want to select "Latest 2.6 Stable" (i386 or x86_64) in your profile, save, and reboot -- in addition to your other cleaning up.

-Chris


OK, I did that. Now I am at a complete loss as to how someone got in. I can see from my Linode performance graphs that in that 23 minute range, there was tons of network activity. And then it all goes dead.

Now, there are exactly two users who have a login shell on my server: myself, and root. root's password is quite strong, as is mine. So I don't think that anyone got in directly via ssh.

I'm suspecting that they must have gotten in via a service like apache (most likely), or dovecot or postfix (less likely). In any case, I think they would have been isolated to running with priveleges of the http, postfix, or dovecot user. Unless they took advantage of the local root exploit you mentioned, but if they had, wouldn't I have seen more activity than I did?

I'm just trying to figure out how bad it's likely to be.

Once years ago I got some kind of Apache virus but because it was limited to the apache user, it was relatively easy to clean out. I wonder if I am in the same situation now, or if my entire host is somehow so compromised that I can't trust root anymore ...


Top
   
 Post subject:
PostPosted: Thu Aug 27, 2009 5:15 pm 
Offline
Senior Member

Joined: Thu Aug 28, 2003 12:57 am
Posts: 273
OK so I've dug around a little, and I just can't see where the exploit could have gotten root access. If it had, I would expect it to remove traces of itself from the logs (if the author was smart), and it didn't. And, I don't see anything else going on with my system that indicates that it was exploited. It's not sending out spam or trying any more ssh attacks.

What I really don't understand is why it stopped after 23 minutes. Why wouldn't it have just kept running indefinitely? Maybe it exhausted its ssh attacks against all other hosts on the local network (sorry everyone!)? And since it didn't have anything else to do without root access, it just quit?


Top
   
 Post subject:
PostPosted: Fri Aug 28, 2009 10:02 am 
Offline
Junior Member

Joined: Tue Dec 09, 2008 2:33 pm
Posts: 49
Website: http://www.ragtop.org
Location: Gilbert, AZ
bji wrote:
Once years ago I got some kind of Apache virus but because it was limited to the apache user, it was relatively easy to clean out. I wonder if I am in the same situation now, or if my entire host is somehow so compromised that I can't trust root anymore ...


If you suspect your host is compromised at all, start over with a fresh install. It might seem like a lot of work, but unless you have a reliable way to tell what has been changed (like tripwire) then you are just asking for more trouble. You have no idea what the attacker could have modified and what type of backdoors he could have put in.

By the way, it looks like "atack" is a program to brute force attack ssh on other machines.

http://translate.google.com/translate?h ... %2F1355990

It appears to also go by the "brute" name:

http://forum.soft32.com/linux/kernel-br ... 57104.html

http://nileshbansal.blogspot.com/2006/0 ... ttack.html


Top
   
 Post subject:
PostPosted: Fri Aug 28, 2009 10:29 am 
Offline
Senior Member
User avatar

Joined: Sun Feb 08, 2004 7:18 pm
Posts: 562
Location: Austin
Thanks for the details and advice, bji. It's kind of a frightening story!


Top
   
 Post subject:
PostPosted: Mon Aug 31, 2009 12:14 pm 
Offline
Senior Member

Joined: Thu Apr 08, 2004 3:24 pm
Posts: 92
ICQ: 3765104
Website: http://www.unixfool.com
Yahoo Messenger: wigglit2001@yahoo.com
Location: VA
I don't think anyone got into your system (unless you can provide more concrete evidence).

I think what may have happened is that someone attempted a DoS or DDoS against your webserver. The conntrack logs support this. Your system's connection table filled up. If there are tons of connections that the system is attempting to track, the system only has finite resources to track connections...if there are a massive amount of connections and the attack is long-lived, your system will eventually run out of resources and crap itself (dunno if your system locked up or recovered on its own).

Typically, we see this with firewalls attempting to track massive infections such as Slammer or Blaster. While the FW itself isn't infected, the FW will eventually bog down and/or crash. Note that this can happen with any type of connection flood...there doesn't have to be a compromise or intrusion for this to happen.

I'd look at your webserver logs (or your SSH logs, since that port was also showing in your netstat output, although I doubt its an SSH attack that happened).


Top
   
 Post subject:
PostPosted: Mon Aug 31, 2009 12:44 pm 
Offline
Senior Member

Joined: Tue Jan 22, 2008 2:10 am
Posts: 103
unixfool, the 'atack' logs indicate that they _did_ get into the system, and ran this 'atack' program. Unless there's a legitimate 'atack' program on the system, there really is a problem here.


Top
   
 Post subject:
PostPosted: Mon Aug 31, 2009 1:45 pm 
Offline
Senior Member

Joined: Thu Apr 08, 2004 3:24 pm
Posts: 92
ICQ: 3765104
Website: http://www.unixfool.com
Yahoo Messenger: wigglit2001@yahoo.com
Location: VA
If there was actually a system breach, there'd a lot more than what the OP posted.

I conduct first-, second-, and third-level attack and compromise analysis for a living. IMO, kernel logs don't always mean what people think they mean (at least regarding compromise). You can't just run some exploit against a kernel without having a vector (avenue of attack, which is more than likely a network service). Kernel logs don't define the endgame of security, unless it is actually a kernel-level attack, which would suggest that someone is logged in locally to the machine (although this may NOT be the case with the OP). conntrack is a kernel-level function of network connectivity, which means that anyone conducting a DoS will affect connection tracking.

You're doing this admin a disservice if you think that what he described is an actual compromise.

Looking at 'atack segfault' under Google shows the following:

http://www.google.com/search?hl=en&q=at ... art=0&sa=N

Almost ALL of those links suggest a denial of service and have similar content as the OP posted. If it quacks and waddles like a duck, its either a relative to the duck or IS a duck, IMO.

Here's a GOOD one (from http://kerneltrap.org/mailarchive/linux ... 23/5223214):

Code:
doing a "ping -f -l 3" on my host towards my board on linus tree as of 
Friday results in lots of:
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
__ratelimit: 11 callbacks suppressed
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.
nf_conntrack: table full, dropping packet.

for ucc_geth on a MPC832x.
This really looks strange to me, ideas?


If this type of activity is directed against an application, there's where the segfault comes in. Regarding the 'atack' tag, I've no idea where that is coming from but to suggest that someone injected that onto the system is to pull suppositions out of thin air. People in my field actually need data to prove that something like this occurred. That's where application logs

All of this is circumstantial, since there are only kernel logs that were provided. Still...

Secondly, there's the issue of him reinstalling (because he was told to do so) without understanding the nature of the attack. There's a chance it may happen, again, within hours/days of his reinstall if he doesn't understand the who/what/where/why of the prior attack. I've seen machine owners reinstall 3-4 times and get compromised 3-4 times afterward...all because the admins didn't understand the underlying issue.


Top
   
 Post subject:
PostPosted: Mon Aug 31, 2009 2:22 pm 
Offline
Junior Member

Joined: Tue Dec 09, 2008 2:33 pm
Posts: 49
Website: http://www.ragtop.org
Location: Gilbert, AZ
All signs point to him being compromised though. The atack program is a ssh cracking tool and part of the unixcod package. You can find it at http://www.bart-design.co.uk/unixcod/

There are several other people that have had similar messages appear in their logs and they were all compromised too. I posted a link to one earlier. Here is another that found atack running in "/var/tmp/ ....... / . /unixcod/atack":

http://ibot.rikers.org/%23debian/20080502.html.gz

While I agree he needs to figure out how he got compromised in order to patch and protect against it in the future, telling him that he was simply DoS'd and to not worry about it is not very good advice either.


Top
   
 Post subject:
PostPosted: Mon Aug 31, 2009 4:07 pm 
Offline
Senior Member

Joined: Fri Dec 07, 2007 1:37 am
Posts: 385
Location: NC, USA
unixfool wrote:
You're doing this admin a disservice if you think that what he described is an actual compromise.

Maybe you missed the strange coincidence that within one second of the OP's segfault log, MY logs (posted earlier) show his system starting an ssh attack against my linode.

Someone got at least user-level access on his system and was looking for more hosts to breach. I don't claim to do this for a living, but if he wasn't compromised then I don't know what the word means.


Top
   
 Post subject:
PostPosted: Mon Aug 31, 2009 5:48 pm 
Offline
Senior Member

Joined: Thu Aug 28, 2003 12:57 am
Posts: 273
Stever wrote:
unixfool wrote:
You're doing this admin a disservice if you think that what he described is an actual compromise.

Maybe you missed the strange coincidence that within one second of the OP's segfault log, MY logs (posted earlier) show his system starting an ssh attack against my linode.

Someone got at least user-level access on his system and was looking for more hosts to breach. I don't claim to do this for a living, but if he wasn't compromised then I don't know what the word means.


I agree - unixfool, I think you must have just missed the part of my logs where I listed the lines indicating that a program named "atack" had run locally on my system (and segfaulted quite alot).

What I think happened is that this atack program forked itself tons of times to issue SSH attacks on other hosts on the network simultaneously. My Linode control panel graphs showed that in this 23 minute time period, my system went to 100% CPU usage, tons of network activity, and all of my memory got used up. I suspect that all of the seg faults are due to fork calls failing when the system was out of memory and the atack program not gracefully handling that.

There are three big questions from my perspective:

1. How did they get in? I have searched for known exploits on the handful of services that I am running, and I can't find anything. So maybe there are exploits in PHP or apache or some other software that just isn't known globally yet but is known to some hackers. Or more likely, I just didn't look hard enough when searching for known exploits.

2. Did they acquire any privileges above and beyond the user process that the atack program ran as? Unfortunately, I can't tell what user ran 'atack'. And unfortunately again, I was apparently running a kernel susceptible to the local root explot that was recently fixed in the newer kernels. And yet, I don't see any evidence of having been rooted. It is possible that some process is sitting there on my system completely hidden from me somehow, but I've done pretty exhaustive searches and I don't find anything. netstat shows nothing. process accounting (via acct, which I installed after this happened) shows nothing. All of the find commands and manual file examination shows nothing. The linode control panel shows that my box isn't suffering the CPU, memory, or network load of running any kind of active explot. I basically have no evidence that the user access was elevated to root access by the attacker. I know, the truly paranoid would say that I shouldn't trust anything and should just re-install. And maybe that is a reasonable level of paranoia. But for the time being, I'm watching and waiting to see if there is ever any other evidence.

3. Is my system still vulnerable? Since I don't know how they got in, I don't know if I've "fixed" the problem. I have installed process accounting to help me watch for any further shenanigans ... any other suggestions for stuff to install to help me watch out for a recurrence of this attack?


Top
   
 Post subject:
PostPosted: Mon Aug 31, 2009 7:15 pm 
Offline
Junior Member

Joined: Wed Sep 03, 2008 9:51 am
Posts: 25
There's a few things you can do that won't guarantee you weren't rooted, but might help in finding out what happened.

Use the find command to find all files created since that day. If there's too much in that output, try limiting it to all executable files created since that day.

Do you use rpm, apt, or something else? With RPM, you can verify all files against the md5sum from the database to see if any binaries were modified from the packaged version. I'm sure apt has a similar feature.

Find and install chkrootkit.

Find all occurences of '.bash_history', and make sure there's nothing funny in any of those.

Many "hackers" using rootkits will patch netstat, but not lsof. Use it to see if there's any ports listening that shoudn't. If it's already installed, remove it, and reinstall it from a binary package.

Hopefully, by this time you have some files to work with. Whatever you do, don't run them. Copy them off to a different host (like a throwaway VM), and inspect them there. File ownership might point you in the direction of which daemon was exploited.

Hope some of that helps!


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic


Who is online

Users browsing this forum: No registered users and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group