Linode Forum
https://forum.linode.com/

server stops every few days
https://forum.linode.com/viewtopic.php?f=19&t=11477		 Page 1 of 2
Author:  john_r_h [ Sun Dec 14, 2014 11:30 am ]
Post subject:  server stops every few days
I hired a contractor, who was highly recommended, build a LAMP stack on my Linode VPS (Linode also highly recommended). Every few days the server becomes un-ping-able, and often after a few hours comes up again.

My contractor says he has done nothing unusual with the setup and that something is on on the Linode itself. Linode Support has implied that the fault is with the server build, and there is nothing more they can, or need to, do since the Linode itself is running. I have not loaded my app or basically even touched the thing except for logging in via Lish to try to read logs.

Any help leading to a real solution would be greatly appreciated. I am willing to make you a full-rights user in my Linode account if it would help, since there is nothing sensitive loaded at this point.

Thanks.
Author:  obs [ Sun Dec 14, 2014 12:18 pm ]
Post subject:  Re: server stops every few days
Are you getting any reboot alerts or other alerts from Linode?

What OS are you running?

What's the contents of /var/log/syslog or /var/log/messages around the time you're having problems?
Author:  john_r_h [ Tue Dec 16, 2014 6:20 am ]
Post subject:  server stops every few days
obs wrote:
Are you getting any reboot alerts or other alerts from Linode?
What OS are you running?
What's the contents of /var/log/syslog or /var/log/messages around the time you're having problems?


Thanks for the quick response.

Getting no Linode alerts (unless I force a reboot, so alerts are working)

OS: CentOS 7

My contractor has checked the logs and says nothing unusual appears. I will begin checking them out myself. I am now running a script (on a different server) that pings my Linode every minute and reports if the status has changed from the previous report.
Author:  masonm [ Tue Dec 16, 2014 8:20 pm ]
Post subject:  Re: server stops every few days
I've had that happen when a server runs out of memory and begins swapping. Run "dmesg" and see if the out-of-memory killer has ever been invoked. What are you using to monitor memory usage (if anything)? Also, are you able to access the server via Lish while it's unpingable?
Author:  john_r_h [ Fri Dec 19, 2014 12:53 pm ]
Post subject:  Re: server stops every few days
masonm wrote:
I've had that happen when a server runs out of memory and begins swapping. Run "dmesg" and see if the out-of-memory killer has ever been invoked. What are you using to monitor memory usage (if anything)? Also, are you able to access the server via Lish while it's unpingable?


Thanks for the response.

I can access via Lish while it is unpingable.

When I run "dmesg" I get a long list of firewall statements about blocking various TCP and UDP hits, but nothing else.

My Linode Dashboard shows large output spike at (what I think is) the moment of crash.

After a crash early this morning (~1 am EST) it would not come back even after three reboots. Somehow it came back by itself since then. A few minutes ago, I entered some false creds in the everheldwebgroup.com/phpmyadmin login box (not an injection attempt, mind you, just a wrong name), and the whole thing went down. Could be a coincidence, but the thought that anyone on the planet can crash my server this easily is unsettling.
Author:  masonm [ Fri Dec 19, 2014 1:26 pm ]
Post subject:  Re: server stops every few days
If you can access it via Lish but not externally then there's clearly a networking issue, most likely involving iptables. Run "ps aux" and "iptables -L -n" as root and post the output of both commands here.
Author:  john_r_h [ Fri Dec 19, 2014 3:04 pm ]
Post subject:  Re: server stops every few days
OK, Took me a while to figure out how to collect all the input, but here it is:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.2 48464 5780 ? Ss 06:13 0:05 /sbin/init nosep nodevfs
root 2 0.0 0.0 0 0 ? S 06:13 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 06:13 0:00 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< 06:13 0:00 [kworker/0:0H]
root 6 0.0 0.0 0 0 ? S 06:13 0:00 [kworker/u4:0]
root 7 0.0 0.0 0 0 ? S 06:13 0:01 [rcu_sched]
root 8 0.0 0.0 0 0 ? S 06:13 0:00 [rcu_bh]
root 9 0.0 0.0 0 0 ? S 06:13 0:00 [migration/0]
root 10 0.0 0.0 0 0 ? S 06:13 0:00 [migration/1]
root 11 0.0 0.0 0 0 ? S 06:13 0:00 [ksoftirqd/1]
root 13 0.0 0.0 0 0 ? S< 06:13 0:00 [kworker/1:0H]
root 14 0.0 0.0 0 0 ? S< 06:13 0:00 [khelper]
root 15 0.0 0.0 0 0 ? S 06:13 0:00 [kdevtmpfs]
root 16 0.0 0.0 0 0 ? S< 06:13 0:00 [netns]
root 20 0.0 0.0 0 0 ? S 06:13 0:00 [xenwatch]
root 21 0.0 0.0 0 0 ? S 06:13 0:00 [xenbus]
root 211 0.0 0.0 0 0 ? S< 06:13 0:00 [writeback]
root 214 0.0 0.0 0 0 ? S< 06:13 0:00 [crypto]
root 215 0.0 0.0 0 0 ? S< 06:13 0:00 [bioset]
root 216 0.0 0.0 0 0 ? S< 06:13 0:00 [kblockd]
root 227 0.0 0.0 0 0 ? S< 06:13 0:00 [md]
root 318 0.0 0.0 0 0 ? S< 06:13 0:00 [rpciod]
root 440 0.0 0.0 0 0 ? S 06:13 0:00 [kswapd0]
root 510 0.0 0.0 0 0 ? S 06:13 0:00 [fsnotify_mark]
root 516 0.0 0.0 0 0 ? S 06:13 0:00 [ecryptfs-kthrea]
root 518 0.0 0.0 0 0 ? S< 06:13 0:00 [nfsiod]
root 520 0.0 0.0 0 0 ? S< 06:13 0:00 [cifsiod]
root 526 0.0 0.0 0 0 ? S 06:13 0:00 [jfsIO]
root 527 0.0 0.0 0 0 ? S 06:13 0:00 [jfsCommit]
root 529 0.0 0.0 0 0 ? S 06:13 0:00 [jfsCommit]
root 531 0.0 0.0 0 0 ? S 06:13 0:00 [jfsSync]
root 533 0.0 0.0 0 0 ? S< 06:13 0:00 [xfsalloc]
root 535 0.0 0.0 0 0 ? S< 06:13 0:00 [xfs_mru_cache]
root 537 0.0 0.0 0 0 ? S< 06:13 0:00 [xfslogd]
root 543 0.0 0.0 0 0 ? S< 06:13 0:00 [glock_workqueue]
root 545 0.0 0.0 0 0 ? S< 06:13 0:00 [delete_workqueu]
root 555 0.0 0.0 0 0 ? S< 06:13 0:00 [gfs_recovery]
root 1150 0.0 0.0 0 0 ? S 06:13 0:00 [khvcd]
root 1247 0.0 0.0 0 0 ? S< 06:13 0:00 [bioset]
root 1248 0.0 0.0 0 0 ? S< 06:13 0:00 [drbd-reissue]
root 1264 0.0 0.0 0 0 ? S< 06:13 0:00 [kpsmoused]
root 1267 0.0 0.0 0 0 ? S< 06:13 0:00 [raid5wq]
root 1272 0.0 0.0 0 0 ? S< 06:13 0:00 [dm_bufio_cache]
root 1297 0.0 0.0 0 0 ? S< 06:13 0:00 [ipv6_addrconf]
root 1317 0.0 0.0 0 0 ? S< 06:13 0:00 [bioset]
root 1345 0.0 0.0 0 0 ? S< 06:13 0:00 [deferwq]
root 1348 0.0 0.0 0 0 ? S< 06:13 0:00 [reiserfs/xvda]
root 1349 0.0 0.0 0 0 ? S< 06:13 0:00 [kworker/0:1H]
root 1350 0.0 0.0 0 0 ? S 06:13 0:00 [jbd2/xvda-8]
root 1351 0.0 0.0 0 0 ? S< 06:13 0:00 [ext4-rsv-conver]
root 1373 0.0 0.4 42996 8212 ? Ss 06:13 0:02 /usr/lib/systemd/systemd-journald
root 1374 0.0 0.0 0 0 ? S 06:13 0:00 [kauditd]
root 1477 0.0 0.1 40720 3116 ? Ss 06:13 0:00 /usr/lib/systemd/systemd-udevd
root 1769 0.0 0.1 116676 3200 ? S<sl 06:13 0:00 /sbin/auditd -n
avahi 2066 0.0 0.1 28080 2472 ? Ss 06:13 0:00 avahi-daemon: running [server.local]
root 2070 0.0 0.7 535972 14612 ? Ssl 06:13 0:05 /usr/sbin/NetworkManager --no-daemon
root 2074 0.0 0.4 207992 8316 ? Ssl 06:13 0:00 /usr/sbin/rsyslogd -n
root 2076 0.0 0.9 549980 20184 ? Ssl 06:13 0:06 /usr/bin/python -Es /usr/sbin/tuned -l -P
avahi 2079 0.0 0.0 27948 220 ? S 06:13 0:00 avahi-daemon: chroot helper
dbus 2084 0.0 0.1 26700 3020 ? Ss 06:13 0:04 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root 2085 0.0 0.1 34688 2980 ? Ss 06:13 0:02 /usr/lib/systemd/systemd-logind
chrony 2087 0.0 0.1 24728 2524 ? S 06:13 0:00 /usr/sbin/chronyd -u chrony
root 2090 0.0 0.1 126304 3032 ? Ss 06:13 0:00 /usr/sbin/crond -n
root 2106 0.0 0.0 110008 1828 tty1 Ss+ 06:13 0:00 /sbin/agetty --noclear tty1
root 2118 0.0 0.0 6488 124 ? Ss 06:13 0:01 /sbin/iprupdate --daemon
root 2125 0.0 0.0 6488 124 ? Ss 06:13 0:01 /sbin/iprinit --daemon
polkitd 2359 0.0 0.5 513848 11512 ? Ssl 06:13 0:01 /usr/lib/polkit-1/polkitd --no-debug
root 2360 0.0 0.0 39128 92 ? Ss 06:13 0:00 /sbin/iprdump --daemon
root 2418 0.0 0.9 107248 19060 ? S 06:13 0:00 /sbin/dhclient -d -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-5fbad332-e2ed-4c3d-bd30-44e3507a717c-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0
mysql 2605 0.0 0.1 115348 3168 ? Ss 06:13 0:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
root 2611 0.0 0.9 260384 18440 ? Ss 06:13 0:03 php-fpm: master process (/usr/local/php56/etc/php-fpm.conf)
root 2618 0.0 0.0 19764 1904 ? Ss 06:13 0:01 /usr/local/directadmin/da-popb4smtp
nobody 2619 0.0 0.2 64692 5168 ? Ss 06:13 0:00 /usr/local/directadmin/directadmin d
root 2628 0.0 0.1 152840 3904 ? Ss 06:13 0:00 pure-ftpd (SERVER)
root 2637 0.0 0.2 82796 6100 ? Ss 06:13 0:00 /usr/sbin/sshd -D
named 2658 0.0 1.0 240448 21352 ? Ssl 06:13 0:00 /usr/sbin/named -u named
nobody 2690 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d
nobody 2691 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d
nobody 2692 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d
nobody 2693 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d
nobody 2694 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d
nobody 2695 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d
nobody 2696 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d
nobody 2698 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d
nobody 2699 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d
nobody 2700 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d
mysql 2743 0.0 4.4 697320 91084 ? Sl 06:13 0:36 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/lib/mysql/server.everheldwebgroup.com.err --pid-file=server.everheldwebgroup.com.pid
root 2748 0.0 0.9 165696 18704 ? Ss 06:13 0:29 lfd - sleeping
root 2808 0.0 0.3 67620 7580 ? Ss 06:13 0:02 /usr/sbin/httpd -k start
apache 2879 0.0 0.5 1332260 11568 ? Sl 06:13 0:24 /usr/sbin/httpd -k start
apache 2880 0.0 0.6 1331644 12856 ? Sl 06:13 0:24 /usr/sbin/httpd -k start
root 2955 0.0 0.1 18664 2500 ? Ss 06:13 0:00 /usr/sbin/dovecot -F
dovecot 2959 0.0 0.2 48172 5592 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2960 0.0 0.2 48176 5508 ? S 06:13 0:00 dovecot/imap-login
dovecot 2961 0.0 0.1 12364 2232 ? S 06:13 0:00 dovecot/anvil [33 connections]
root 2962 0.0 0.1 12496 2364 ? S 06:13 0:00 dovecot/log
dovecot 2964 0.0 0.2 48172 5516 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2965 0.0 0.2 48172 5532 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2966 0.0 0.2 48172 5556 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2967 0.0 0.2 48172 5532 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2968 0.0 0.2 48172 5552 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2969 0.0 0.2 48172 5536 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2970 0.0 0.2 48172 5552 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2971 0.0 0.2 48172 5576 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2972 0.0 0.2 48172 5548 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2973 0.0 0.2 48172 5544 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2974 0.0 0.2 48176 5588 ? S 06:13 0:00 dovecot/imap-login
dovecot 2975 0.0 0.2 48176 5552 ? S 06:13 0:00 dovecot/imap-login
dovecot 2976 0.0 0.2 48176 5540 ? S 06:13 0:00 dovecot/imap-login
dovecot 2977 0.0 0.2 48176 5556 ? S 06:13 0:00 dovecot/imap-login
dovecot 2978 0.0 0.2 48176 5524 ? S 06:13 0:00 dovecot/imap-login
dovecot 2979 0.0 0.2 48176 5560 ? S 06:13 0:00 dovecot/imap-login
dovecot 2980 0.0 0.2 48176 5564 ? S 06:13 0:00 dovecot/imap-login
dovecot 2981 0.0 0.2 48176 5692 ? S 06:13 0:00 dovecot/imap-login
dovecot 2982 0.0 0.2 48176 5560 ? S 06:13 0:00 dovecot/imap-login
dovecot 2983 0.0 0.2 48176 5696 ? S 06:13 0:00 dovecot/imap-login
root 2984 0.0 0.1 15460 3216 ? S 06:13 0:00 dovecot/config
dovecot 2985 0.0 0.2 48172 5540 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2986 0.0 0.2 48172 5516 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2987 0.0 0.2 48172 5560 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2988 0.0 0.2 48172 5540 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2989 0.0 0.2 48172 5560 ? S 06:13 0:00 dovecot/pop3-login
dovecot 2990 0.0 0.2 48176 5696 ? S 06:13 0:00 dovecot/imap-login
dovecot 2991 0.0 0.2 48176 5560 ? S 06:13 0:00 dovecot/imap-login
dovecot 2992 0.0 0.2 48176 5536 ? S 06:13 0:00 dovecot/imap-login
dovecot 2993 0.0 0.2 48176 5532 ? S 06:13 0:00 dovecot/imap-login
dovecot 2994 0.0 0.2 48176 5544 ? S 06:13 0:00 dovecot/imap-login
root 2995 0.0 0.1 17836 3392 ? S 06:13 0:00 dovecot/auth [0 wait, 0 passdb, 0 userdb]
mail 3010 0.0 0.2 62944 5792 ? Ss 06:14 0:00 /usr/sbin/exim -bd -q1h
root 5257 0.0 0.0 0 0 ? S 09:17 0:00 [kworker/u4:2]
root 7233 0.0 0.0 0 0 ? S< 12:05 0:00 [kworker/1:1H]
root 12080 0.0 0.0 0 0 ? S 18:01 0:00 [kworker/1:3]
root 12499 0.0 0.0 0 0 ? S 18:19 0:00 [kworker/0:2]
root 12758 0.0 0.0 0 0 ? S 18:35 0:00 [kworker/1:0]
root 12889 0.0 0.2 188816 4376 ? Ss 18:40 0:00 login -- admin
root 12897 0.0 0.0 0 0 ? S 18:41 0:00 [kworker/0:0]
root 12912 0.0 0.0 0 0 ? S 18:42 0:00 [kworker/1:1]
admin 12923 0.0 0.1 115352 3340 hvc0 Ss 18:42 0:00 -bash
root 12947 0.0 0.2 180544 4116 hvc0 S 18:42 0:00 su
root 12948 0.0 0.1 115352 3408 hvc0 S 18:42 0:00 bash
root 13081 0.0 0.1 123360 2544 hvc0 R+ 18:48 0:00 ps aux

...and from iptables:

Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT tcp -- 207.192.69.5 0.0.0.0/0 tcp dpt:53
ACCEPT udp -- 207.192.69.5 0.0.0.0/0 udp dpt:53
ACCEPT tcp -- 207.192.69.5 0.0.0.0/0 tcp spt:53
ACCEPT udp -- 207.192.69.5 0.0.0.0/0 udp spt:53
ACCEPT tcp -- 207.192.69.4 0.0.0.0/0 tcp dpt:53
ACCEPT udp -- 207.192.69.4 0.0.0.0/0 udp dpt:53
ACCEPT tcp -- 207.192.69.4 0.0.0.0/0 tcp spt:53
ACCEPT udp -- 207.192.69.4 0.0.0.0/0 udp spt:53
ACCEPT tcp -- 97.107.133.4 0.0.0.0/0 tcp dpt:53
ACCEPT udp -- 97.107.133.4 0.0.0.0/0 udp dpt:53
ACCEPT tcp -- 97.107.133.4 0.0.0.0/0 tcp spt:53
ACCEPT udp -- 97.107.133.4 0.0.0.0/0 udp spt:53
LOCALINPUT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
INVALID tcp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:20
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:21
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:25
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:53
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:67
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:110
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:143
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:443
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:465
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:587
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:993
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:995
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:2222
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:1857
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:20
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:21
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:53
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 8 limit: avg 1/sec burst 5
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 0 limit: avg 1/sec burst 5
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 11
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 3
LOGDROPIN all -- 0.0.0.0/0 0.0.0.0/0

Chain FORWARD (policy DROP)
target prot opt source destination

Chain OUTPUT (policy DROP)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 207.192.69.5 tcp dpt:53
ACCEPT udp -- 0.0.0.0/0 207.192.69.5 udp dpt:53
ACCEPT tcp -- 0.0.0.0/0 207.192.69.5 tcp spt:53
ACCEPT udp -- 0.0.0.0/0 207.192.69.5 udp spt:53
ACCEPT tcp -- 0.0.0.0/0 207.192.69.4 tcp dpt:53
ACCEPT udp -- 0.0.0.0/0 207.192.69.4 udp dpt:53
ACCEPT tcp -- 0.0.0.0/0 207.192.69.4 tcp spt:53
ACCEPT udp -- 0.0.0.0/0 207.192.69.4 udp spt:53
ACCEPT tcp -- 0.0.0.0/0 97.107.133.4 tcp dpt:53
ACCEPT udp -- 0.0.0.0/0 97.107.133.4 udp dpt:53
ACCEPT tcp -- 0.0.0.0/0 97.107.133.4 tcp spt:53
ACCEPT udp -- 0.0.0.0/0 97.107.133.4 udp spt:53
LOCALOUTPUT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:53
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp spt:53
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:53
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
INVALID tcp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:20
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:21
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:25
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:53
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:67
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:110
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:113
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:443
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:2222
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:1857
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:20
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:21
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:53
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:113
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:123
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 0
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 8
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 11
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 3
LOGDROPOUT all -- 0.0.0.0/0 0.0.0.0/0

Chain ALLOWIN (1 references)
target prot opt source destination
ACCEPT all -- 71.174.180.0/24 0.0.0.0/0
ACCEPT all -- 71.174.180.183 0.0.0.0/0
ACCEPT all -- 134.42.112.2 0.0.0.0/0

Chain ALLOWOUT (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 71.174.180.0/24
ACCEPT all -- 0.0.0.0/0 71.174.180.183
ACCEPT all -- 0.0.0.0/0 134.42.112.2

Chain DENYIN (1 references)
target prot opt source destination
DROP all -- 76.109.51.121 0.0.0.0/0
DROP all -- 79.188.233.110 0.0.0.0/0
DROP all -- 37.233.38.46 0.0.0.0/0
DROP all -- 61.19.253.26 0.0.0.0/0
DROP all -- 77.241.99.130 0.0.0.0/0
DROP all -- 185.61.136.111 0.0.0.0/0
DROP all -- 88.211.134.50 0.0.0.0/0
DROP all -- 94.102.52.186 0.0.0.0/0
DROP all -- 85.238.127.45 0.0.0.0/0
DROP all -- 173.166.245.5 0.0.0.0/0
DROP all -- 62.33.192.25 0.0.0.0/0
DROP all -- 72.93.39.7 0.0.0.0/0
DROP all -- 50.176.69.14 0.0.0.0/0

Chain DENYOUT (1 references)
target prot opt source destination
LOGDROPOUT all -- 0.0.0.0/0 76.109.51.121
LOGDROPOUT all -- 0.0.0.0/0 79.188.233.110
LOGDROPOUT all -- 0.0.0.0/0 37.233.38.46
LOGDROPOUT all -- 0.0.0.0/0 61.19.253.26
LOGDROPOUT all -- 0.0.0.0/0 77.241.99.130
LOGDROPOUT all -- 0.0.0.0/0 185.61.136.111
LOGDROPOUT all -- 0.0.0.0/0 88.211.134.50
LOGDROPOUT all -- 0.0.0.0/0 94.102.52.186
LOGDROPOUT all -- 0.0.0.0/0 85.238.127.45
LOGDROPOUT all -- 0.0.0.0/0 173.166.245.5
LOGDROPOUT all -- 0.0.0.0/0 62.33.192.25
LOGDROPOUT all -- 0.0.0.0/0 72.93.39.7
LOGDROPOUT all -- 0.0.0.0/0 50.176.69.14

Chain INVALID (2 references)
target prot opt source destination
INVDROP all -- 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x03/0x03
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x06/0x06
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x05/0x05
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x11/0x01
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x18/0x08
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x30/0x20
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:!0x17/0x02 ctstate NEW

Chain INVDROP (10 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0

Chain LOCALINPUT (1 references)
target prot opt source destination
ALLOWIN all -- 0.0.0.0/0 0.0.0.0/0
DENYIN all -- 0.0.0.0/0 0.0.0.0/0

Chain LOCALOUTPUT (1 references)
target prot opt source destination
ALLOWOUT all -- 0.0.0.0/0 0.0.0.0/0
DENYOUT all -- 0.0.0.0/0 0.0.0.0/0

Chain LOGDROPIN (1 references)
target prot opt source destination
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:67
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:68
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:68
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:111
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:111
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:113
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:113
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpts:135:139
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpts:135:139
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:445
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:445
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:500
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:500
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:513
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:513
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:520
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:520
LOG tcp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 0 level 4 prefix "Firewall: *TCP_IN Blocked* "
LOG udp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 0 level 4 prefix "Firewall: *UDP_IN Blocked* "
LOG icmp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 0 level 4 prefix "Firewall: *ICMP_IN Blocked* "
DROP all -- 0.0.0.0/0 0.0.0.0/0

Chain LOGDROPOUT (14 references)
target prot opt source destination
LOG tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 30/min burst 5 LOG flags 8 level 4 prefix "Firewall: *TCP_OUT Blocked* "
LOG udp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 8 level 4 prefix "Firewall: *UDP_OUT Blocked* "
LOG icmp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 8 level 4 prefix "Firewall: *ICMP_OUT Blocked* "
DROP all -- 0.0.0.0/0 0.0.0.0/0
Author:  masonm [ Fri Dec 19, 2014 3:19 pm ]
Post subject:  Re: server stops every few days
This iptables rule is likely why your box comes unpingable:
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 8 limit: avg 1/sec burst 5

What that's saying is you can only ping 5 times before it'll start limiting you to 1 a second, blocking anything more than that. That just affects pings though; everything else should work fine. Next time you try pinging, limit it to once every two seconds to make sure you don't get blocked. In Linux, this can be done with "ping -i 2".

I noticed you have login failure daemon ("lfd") running, which is probably what blocked you when you failed to login. Talk to your contractor about tweaking it if you think it's too aggressive.
Author:  john_r_h [ Fri Dec 19, 2014 3:35 pm ]
Post subject:  Re: server stops every few days
masonm wrote:
This iptables rule is likely why your box comes unpingable:
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 8 limit: avg 1/sec burst 5

What that's saying is you can only ping 5 times before it'll start limiting you to 1 a second, blocking anything more than that. That just affects pings though; everything else should work fine. Next time you try pinging, limit it to once every two seconds to make sure you don't get blocked. In Linux, this can be done with "ping -i 2".

I noticed you have login failure daemon ("lfd") running, which is probably what blocked you when you failed to login. Talk to your contractor about tweaking it if you think it's too aggressive.


Thanks for the response.

Regarding ping, I have a cron-script running from a different site that pings this box only oncer per minute, so I don't think I am hitting it too hard.

As for the blocked login, I am not sure what is meant by "blocked you". Ping and phpmyadmin-log-in aside, if you try going to http://everheldwebgroup.com/ nothing is found. If ping and login attempts can result in taking the server offline like this then I am sunk. Right?

Again, thanks for all the help.
Author:  masonm [ Fri Dec 19, 2014 10:42 pm ]
Post subject:  Re: server stops every few days
john_r_h wrote:
Regarding ping, I have a cron-script running from a different site that pings this box only oncer per minute, so I don't think I am hitting it too hard.


Could be pings from someone else. That rule doesn't discrimate based on IP, it applies to all incoming packets.

Quote:
As for the blocked login, I am not sure what is meant by "blocked you".


From http://configserver.com/cp/csf.html:

"To complement the ConfigServer Firewall (csf), we have developed a Login Failure Daemon (lfd) process that runs all the time and periodically (every X seconds) scans the latest log file entries for login attempts against your server that continually fail within a short period of time. Such attempts are often called "Brute-force attacks" and the daemon process responds very quickly to such patterns and blocks offending IP's quickly. "

It's almost certainly using iptables for doing the actual blocking, so the next time you get blocked you should be able to login via Lish and unblock yourself by deleting the iptables rule. Talk to your contractor for more information.

Quote:
If ping and login attempts can result in taking the server offline like this then I am sunk. Right?


No. ICMP echo packets can usually be blocked completely without adversly affecting a server. I'm not saying that's a good idea (since you lose the ability to troubleshoot with ping), but just because a server can't be pinged doesn't mean it's offline.

Also, the lfd blocks are per-IP if I'm reading the docs right, so it only makes the server inaccessible to you, not anyone else.
Author:  john_r_h [ Sat Dec 20, 2014 11:24 am ]
Post subject:  Re: server stops every few days
I do not pretend to understand completely your previous reply, so I will continue to digest (thanks particularly for the link). Seeing phrases like the following:

"...next time you get blocked" and "...it only makes the server inaccessible to you..."

I am starting to think that I lead this discussion in a wrong direction with my mention of my own log-ins.

First, I have been taking care to make sure I am not looking at the site as one user. At my place of work I have access to two entirely separate Internet connections, for employees and visitors, with separate security measures, etc., and absolutely separate Internet-facing IPs. At home, I have accest to two separate DSL connections, different physical connections, with a different carrier than the one at work.

Second, I want to emphasize that my domain becomes unavailable (to all), or goes back online at times when I have not even been looking at it. It all started soon after the Linode went online and well before I put put any auto-ping script in place.

I am not disputing the assertion that it is a networking issue, but I want to make sure I have not mislead.

Here, for what it is worth, is a screenshot of one of my dashboard graphs from this morning (uploaded to my demo site on a VPS other than the Linode in question):

http://everheldwebgroupdemo.com/aa_lino ... shot_1.png

Thanks
Author:  vonskippy [ Sat Dec 20, 2014 3:30 pm ]
Post subject:  Re: server stops every few days
If you're still having problems, I'd try two things:

One, load MONIT so you can monitor what processes are doing what and when - then correlate that info with the time(s) your external ping monitor is saying you're offline.

Two, at least for testing, simplify (a lot) your IPTABLES ruleset, which seems to me, to be a right mess.

Example:

Code:
Chain ALLOWIN (1 references)
target prot opt source destination
ACCEPT all -- 71.174.180.0/24 0.0.0.0/0
ACCEPT all -- 71.174.180.183 0.0.0.0/0
ACCEPT all -- 134.42.112.2 0.0.0.0/0


Line 2 is completely superfluous, since in Line 1 you've already allowed that entire /24 subnet.

In several places, you allow several single IP's only to ALLOW ALL a few lines later.

Not sure what generated that IPTABLES ruleset, but I'd try again.
Author:  john_r_h [ Sat Dec 20, 2014 7:40 pm ]
Post subject:  Re: server stops every few days
OK! So after reading the words "firewall" and "iptables" often enough from you all, I tried just shutting off the firewall, and, bam!, immediately I can get to the site! (Took me a while just to find the command. Seems that CentOS 7 is something of a different animal from previous versions). So I guess I need to get my contractor (or someone) review the settings.

It is wonderful to at least have a direction. Thanks again to all three of you.
Author:  john_r_h [ Tue Dec 23, 2014 12:23 am ]
Post subject:  Re: server stops every few days
So I looked at little more at the iptables I posted above, and, hey, 50.176.69.14 is one of the IPs from which I come to the server! I am sure my contractor did not just write that "deny" line in explicitly. I now think I was wrong to say the server is going done. Instead, it seems to block my IP. If so, it has happened to both my home and place of work, then goes back to accepting me for a while after I reboot the VPS.

What is going on here?! Its like the thing is stalking me.
Author:  Vance [ Tue Dec 23, 2014 4:53 am ]
Post subject:  Re: server stops every few days
It is likely that these firewall rules are added by the ConfigServer Firewall/Login Failure Daemon that masonm described. Somewhere in the logs for that application it should say why your IP was blocked. Often this is because of failed login attempts, or even the number of attempts (successful or not) in a given time period. It all depends on what triggers are set in the application.
Page 1 of 2 All times are UTC-04:00
Powered by phpBB® Forum Software © phpBB Group
http://www.phpbb.com/