Linode Forum
Linode Community Forums
 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Post new topic  Reply to topic
Author Message
PostPosted: Sat Oct 26, 2013 3:27 pm 
Offline
Senior Member

Joined: Wed May 13, 2009 1:32 pm
Posts: 737
Location: Italy
As title.
I'm experiecing stability issues since some months.

The problems occurs really randomly at a distance of 20 to 30 days, but it is too random to say. In any case it occurs rarely, not often but it occurs and this is terrible for me.

I'm using CentOS 6.4 with the latest CentOS kernel using pv-grub.

Since this problems occured many times I'm loggin every five minutes the resources usages, this is the usage on the freezed vps.
Quote:
btmp begins Wed Oct 23 10:04:08 2013
total used free shared buffers cached
Mem: 1015568 945728 69840 0 123092 526536
-/+ buffers/cache: 296100 719468
Swap: 262136 496 261640
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1392 2.2 1.0 2399220 10864 ? Sl Oct22 136:25 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x
502 25965 0.3 6.0 168132 61352 ? S 17:19 0:31 dovecot/imap
mysql 1245 0.1 2.2 771976 22372 ? Sl Oct22 8:52 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
502 27180 0.1 5.1 161032 51868 ? S 18:34 0:06 dovecot/imap
root 1 0.0 0.1 19360 1412 ? Ss Oct22 0:02 /sbin/init
root 2 0.0 0.0 0 0 ? S Oct22 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Oct22 0:00 [migration/0]
root 4 0.0 0.0 0 0 ? S Oct22 0:00 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S Oct22 0:00 [migration/0]
root 6 0.0 0.0 0 0 ? S Oct22 0:01 [watchdog/0]
root 7 0.0 0.0 0 0 ? S Oct22 0:00 [migration/1]
root 8 0.0 0.0 0 0 ? S Oct22 0:00 [migration/1]
root 9 0.0 0.0 0 0 ? S Oct22 0:01 [ksoftirqd/1]
root 10 0.0 0.0 0 0 ? S Oct22 0:01 [watchdog/1]
root 11 0.0 0.0 0 0 ? S Oct22 0:00 [migration/2]
root 12 0.0 0.0 0 0 ? S Oct22 0:00 [migration/2]


As you can see there is not CPU problem and no memory problems.

When the systems locks, it stops responding to SSH, HTTP, email servers, the systems seems dead.
The only things that works perfectly is the lish linode console, this makes me thinking at something strange.

What do you think? What could be the problem? Can I do something more to troubleshoot the problem?

Thanks.


Top
   
PostPosted: Sat Oct 26, 2013 3:30 pm 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
Nothing suspicious on the console like OOM messages?

Is fail2ban doing something funky with your iptable rules, and blocking all network traffic?

If you can access the system via Lish during these times, I'd make sure your iptable rules aren't wonked. Does networking work at all during these times? Outbound?

-Chris


Top
   
PostPosted: Sat Oct 26, 2013 4:33 pm 
Offline
Senior Member

Joined: Wed May 13, 2009 1:32 pm
Posts: 737
Location: Italy
caker wrote:
Nothing suspicious on the console like OOM messages?

Is fail2ban doing something funky with your iptable rules, and blocking all network traffic?

If you can access the system via Lish during these times, I'd make sure your iptable rules aren't wonked. Does networking work at all during these times? Outbound?

-Chris


quite honored to get an answer from caker directly, not joking, I'm honest, so thanks for the reply :)

No, no OOM message on the console, nothing strange, no warnings.
I don't tried to see if my server can communicate with the external during the hangs, now I rebooted it and can't check.

I never checked the iptables rules while the system stopped responding to external IPs.
Do you think that fail2ban is banning all the external IPs?
I don't have any fail2ban logs but fail2ban sends me an email every time it ban an IP, and I don't received any email.

I created a cron that logs every 5 minutes this commands in the hope to understand why the system hangs.
Code:
#!/bin/ksh -p

mkdir -p /root/log_for_crash_detect/day_$(date +%Y-%m-%d)
LOG=/root/log_for_crash_detect/day_$(date +%Y-%m-%d)/log_$(date +%Y-%m-%d-%H-%M)

{
  echo -- date --------------------
  date
  echo -- date-end ----------------
  echo
  echo
  echo -- uptime ------------------
  uptime
  echo -- uptime-end --------------
  echo
  echo
  echo -- last --------------------
  last
  echo -- last-end ----------------
  echo
  echo
  echo -- lastlog -----------------
  lastlog
  echo -- lastlog-end--------------
  echo
  echo
  echo -- lastb -------------------
  lastb
  echo -- lastb-end----------------
  echo
  echo
  echo -- free --------------------
  free
  echo -- free-end ----------------
  echo
  echo
  echo -- ps aux --sort '-pcpu' ---
  ps aux --sort '-pcpu'
  echo -- ps aux ------------------
  echo
  echo
  echo -- iostat 1 5 ---------------------------------------------------------------------
  iostat 1 5
  echo -- iostat 1 5 end -----------------------------------------------------------------
  echo
  echo
  echo -- vmstat 1 5 ---------------------------------------------------------------------
  vmstat 1 5
  echo -- vmstat 1 5 end -----------------------------------------------------------------
  echo
  echo
  echo "-- ps auxf | sort -nr -k 4 | head -5 -----------------------------------------------"
  ps auxf | sort -nr -k 4 | head -5
  echo "-- ps auxf | sort -nr -k 4 | head -5 end -------------------------------------------"
  echo
  echo
  echo "-- ps auxf | sort -nr -k 3 | head -5 -----------------------------------------------"
  ps auxf | sort -nr -k 3 | head -5
  echo "-- ps auxf | sort -nr -k 3 | head -5 end -------------------------------------------"
  echo
  echo
  echo "-- [PID] [30EM] [PATH] && ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20 -------"
  echo [PID] [30EM] [PATH] && ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20
  echo "-- echo [PID] [30EM] [PATH] && ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20 end ----"
  echo
  echo
  echo "-- ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10 -----------------------------"
  ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10
  echo "-- ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10 end -------------------------"
  echo
  echo
  echo "-- iptables -L ---------------------------------------------------------------------"
  iptables -L
  echo "-- iptables -L end -----------------------------------------------------------------"
  echo
  echo

} >> $LOG



Top
   
PostPosted: Sat Oct 26, 2013 6:42 pm 
Offline
Senior Member

Joined: Sun May 23, 2010 1:57 pm
Posts: 315
Website: http://www.jebblue.net
Nope, things are good for me, if I were experiencing a problem with my Linode, I would file a Ticket with Support.


Top
   
PostPosted: Sat Oct 26, 2013 6:47 pm 
Offline
Junior Member
User avatar

Joined: Fri May 04, 2012 8:57 pm
Posts: 49
next time you see this, log into LISH and grab the outputs of the following commands:

iptables -L -n -v
ifconfig

sounds like a firewall issue imo

_________________
me | voltaireMC


Top
   
PostPosted: Sun Oct 27, 2013 5:53 am 
Offline
Senior Member

Joined: Wed May 13, 2009 1:32 pm
Posts: 737
Location: Italy
kbar wrote:
next time you see this, log into LISH and grab the outputs of the following commands:

iptables -L -n -v
ifconfig

sounds like a firewall issue imo


I will add this commands to the "log every five minutes" lists and I will see the output when the problem will occur again.

Thanks for the help.


Top
   
PostPosted: Sun Oct 27, 2013 9:27 am 
Offline
Senior Member
User avatar

Joined: Wed Mar 17, 2004 4:11 pm
Posts: 554
Website: http://www.unixtastic.com
Location: Europe
Fail2ban. The cause of oh so much pain and oh so little benefit.


Top
   
PostPosted: Sun Oct 27, 2013 10:38 am 
Offline
Senior Member

Joined: Wed May 13, 2009 1:32 pm
Posts: 737
Location: Italy
sednet wrote:
Fail2ban. The cause of oh so much pain and oh so little benefit.


how to ban brute force attack against a webmail without fail2ban?


Top
   
PostPosted: Sun Oct 27, 2013 10:42 am 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
sblantipodi wrote:
sednet wrote:
Fail2ban. The cause of oh so much pain and oh so little benefit.


how to ban brute force attack against a webmail without fail2ban?

You can't. A distribute brute force attack renders fail2ban useless.

Personal I don't see the point of fail2ban. Your passwords are either secure or they're not. fail2ban has little benefit other than reducing log spam, imo.

-Chris


Top
   
PostPosted: Sun Oct 27, 2013 11:16 am 
Offline
Senior Member

Joined: Wed May 13, 2009 1:32 pm
Posts: 737
Location: Italy
caker wrote:
sblantipodi wrote:
sednet wrote:
Fail2ban. The cause of oh so much pain and oh so little benefit.


how to ban brute force attack against a webmail without fail2ban?

You can't. A distribute brute force attack renders fail2ban useless.

Personal I don't see the point of fail2ban. Your passwords are either secure or they're not. fail2ban has little benefit other than reducing log spam, imo.

-Chris


never received a distribuite brute force attack (I'm lucky ;) ) but received many attacks who hitted my webmail/postfix for days.
with fail2ban days decreased to three attemps and logs decreased from hundred of MB to dozens of MB.

I'm sure that fail2ban is not the definitive security tool but I find it useful.


Top
   
PostPosted: Sun Oct 27, 2013 11:21 am 
Offline
Senior Member

Joined: Wed May 13, 2009 1:32 pm
Posts: 737
Location: Italy
I add something more if it can help troubleshooting.
when the server stopped responding in the hurry of lift it I read the logs,
no strange attemp against my server, than I restarted iptables.

The system hanged in the restart process of iptables.


Top
   
PostPosted: Sun Oct 27, 2013 11:35 am 
Offline
Senior Member

Joined: Sun May 23, 2010 1:57 pm
Posts: 315
Website: http://www.jebblue.net
sednet wrote:
Fail2ban. The cause of oh so much pain and oh so little benefit.


So stopping people who are actively trying to break into your server is a bad idea? That's what it does, monitors logs, sees suspicious activity, bans the IP. How's that "little benefit"?


Top
   
PostPosted: Sun Oct 27, 2013 11:51 pm 
Offline
Senior Member

Joined: Mon Aug 29, 2011 2:34 am
Posts: 77
jebblue wrote:
So stopping people who are actively trying to break into your server is a bad idea? That's what it does, monitors logs, sees suspicious activity, bans the IP. How's that "little benefit"?


See caker's point. Your password and/or SSH key are either secure or they aren't. fail2ban won't stop a determined attacker from getting into your system if the means of authentication have been compromised. The only questionably useful purpose of fail2ban is to reduce the amount of logspam received from people attempting to brute force a properly secured system. Other measures can be taken to reduce the logspam that can't result in you getting locked out of your own system because you've forgotten your password or have too many SSH keys in your local SSH agent (this can be rather puzzling). Such measures include configuring your logging daemon to not log the messages generated by brute force attempts in the first place, or properly configuring logrotate to rotate and compress old logs. Other options include port knocking and single packet authorization (the latter being the preferred method, as it further validates that you are who you say you are, and supplements existing security).

-Doug


Top
   
PostPosted: Mon Oct 28, 2013 12:55 am 
Offline
Senior Member

Joined: Sun May 23, 2010 1:57 pm
Posts: 315
Website: http://www.jebblue.net
>> can't result in you getting locked out of your own system because

If you can successfully fail to log into your system more than a few times in a row then you might want to consider installing fail2ban or at least iptables rate limiting. Preferably both.


Top
   
PostPosted: Mon Oct 28, 2013 1:54 am 
Offline
Senior Member

Joined: Mon Aug 29, 2011 2:34 am
Posts: 77
jebblue wrote:
>> can't result in you getting locked out of your own system because

If you can successfully fail to log into your system more than a few times in a row then you might want to consider installing fail2ban or at least iptables rate limiting. Preferably both.


Both of which only reduce logspam, and we're back to the point I gave before. Neither of them increase security in any useful way, but instead cause unnecessary headache for legitimate users.

-Doug


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic


Who is online

Users browsing this forum: No registered users and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group