caker wrote:
Nothing suspicious on the console like OOM messages?
Is fail2ban doing something funky with your iptable rules, and blocking all network traffic?
If you can access the system via Lish during these times, I'd make sure your iptable rules aren't wonked. Does networking work at all during these times? Outbound?
-Chris
quite honored to get an answer from caker directly, not joking, I'm honest, so thanks for the reply

No, no OOM message on the console, nothing strange, no warnings.
I don't tried to see if my server can communicate with the external during the hangs, now I rebooted it and can't check.
I never checked the iptables rules while the system stopped responding to external IPs.
Do you think that fail2ban is banning all the external IPs?
I don't have any fail2ban logs but fail2ban sends me an email every time it ban an IP, and I don't received any email.
I created a cron that logs every 5 minutes this commands in the hope to understand why the system hangs.
Code:
#!/bin/ksh -p
mkdir -p /root/log_for_crash_detect/day_$(date +%Y-%m-%d)
LOG=/root/log_for_crash_detect/day_$(date +%Y-%m-%d)/log_$(date +%Y-%m-%d-%H-%M)
{
echo -- date --------------------
date
echo -- date-end ----------------
echo
echo
echo -- uptime ------------------
uptime
echo -- uptime-end --------------
echo
echo
echo -- last --------------------
last
echo -- last-end ----------------
echo
echo
echo -- lastlog -----------------
lastlog
echo -- lastlog-end--------------
echo
echo
echo -- lastb -------------------
lastb
echo -- lastb-end----------------
echo
echo
echo -- free --------------------
free
echo -- free-end ----------------
echo
echo
echo -- ps aux --sort '-pcpu' ---
ps aux --sort '-pcpu'
echo -- ps aux ------------------
echo
echo
echo -- iostat 1 5 ---------------------------------------------------------------------
iostat 1 5
echo -- iostat 1 5 end -----------------------------------------------------------------
echo
echo
echo -- vmstat 1 5 ---------------------------------------------------------------------
vmstat 1 5
echo -- vmstat 1 5 end -----------------------------------------------------------------
echo
echo
echo "-- ps auxf | sort -nr -k 4 | head -5 -----------------------------------------------"
ps auxf | sort -nr -k 4 | head -5
echo "-- ps auxf | sort -nr -k 4 | head -5 end -------------------------------------------"
echo
echo
echo "-- ps auxf | sort -nr -k 3 | head -5 -----------------------------------------------"
ps auxf | sort -nr -k 3 | head -5
echo "-- ps auxf | sort -nr -k 3 | head -5 end -------------------------------------------"
echo
echo
echo "-- [PID] [30EM] [PATH] && ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20 -------"
echo [PID] [30EM] [PATH] && ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20
echo "-- echo [PID] [30EM] [PATH] && ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20 end ----"
echo
echo
echo "-- ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10 -----------------------------"
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10
echo "-- ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10 end -------------------------"
echo
echo
echo "-- iptables -L ---------------------------------------------------------------------"
iptables -L
echo "-- iptables -L end -----------------------------------------------------------------"
echo
echo
} >> $LOG