I checks syslog after a fresh crash this afternoon. I see that oom-killer kicked in
Code:
Feb 10 15:27:13 mylinode kernel: apache2 invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
The previous entry was logged 2 minutes prior
Code:
Feb 10 15:25:01 mylinode CRON[5655]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
I don't think this is the issue though, because if I'm reading the log file correctly, it looks like I have an absurd amount of Apache2 processes running. Monitoring my server using 'top', I typically only see 1-5 Apache processes running at a time.
Code:
Feb 10 15:27:13 mylinode kernel: 16126 pages reserved
Feb 10 15:27:13 mylinode kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Feb 10 15:27:13 mylinode kernel: [ 1456] 0 1456 5071 1 14 273 0 upstart-udev-br
Feb 10 15:27:13 mylinode kernel: [ 1461] 0 1461 12325 1 27 145 -1000 systemd-udevd
Feb 10 15:27:13 mylinode kernel: [ 2444] 0 2444 2560 0 6 573 0 dhclient
Feb 10 15:27:13 mylinode kernel: [ 2649] 102 2649 9781 1 23 88 0 dbus-daemon
Feb 10 15:27:13 mylinode kernel: [ 2727] 0 2727 10865 13 27 69 0 systemd-logind
Feb 10 15:27:13 mylinode kernel: [ 2736] 0 2736 15343 28 32 142 -1000 sshd
Feb 10 15:27:13 mylinode kernel: [ 2738] 0 2738 4787 0 13 43 0 atd
Feb 10 15:27:13 mylinode kernel: [ 2739] 0 2739 5916 23 16 43 0 cron
Feb 10 15:27:13 mylinode kernel: [ 2753] 101 2753 63963 77 27 89 0 rsyslogd
Feb 10 15:27:13 mylinode kernel: [ 2817] 0 2817 3920 1 13 146 0 upstart-file-br
Feb 10 15:27:13 mylinode kernel: [ 2819] 0 2819 4015 0 13 263 0 upstart-socket-
Feb 10 15:27:13 mylinode kernel: [ 2821] 105 2821 472444 47076 358 87305 0 mysqld
Feb 10 15:27:13 mylinode kernel: [ 2867] 0 2867 24659 53 47 390 0 sendmail-mta
Feb 10 15:27:13 mylinode kernel: [ 3158] 0 3158 78253 2750 90 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 3187] 0 3187 66396 1780 31 0 0 fail2ban-server
Feb 10 15:27:13 mylinode kernel: [ 3229] 0 3229 3199 40 11 0 0 getty
Feb 10 15:27:13 mylinode kernel: [ 3345] 103 3345 7863 150 20 0 0 ntpd
Feb 10 15:27:13 mylinode kernel: [ 3368] 0 3368 26446 273 53 0 0 sshd
Feb 10 15:27:13 mylinode kernel: [ 3404] 1000 3404 26446 271 52 0 0 sshd
Feb 10 15:27:13 mylinode kernel: [ 3405] 1000 3405 5563 420 16 0 0 bash
Feb 10 15:27:13 mylinode kernel: [ 4307] 33 4307 80672 12282 109 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4343] 33 4343 116884 48398 180 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4433] 33 4433 84282 16034 117 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4440] 33 4440 83661 16264 116 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4457] 33 4457 81064 12773 110 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4458] 33 4458 83032 14720 114 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4466] 33 4466 87206 18817 121 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4503] 33 4503 113967 46300 178 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4505] 33 4505 85272 16833 118 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4512] 33 4512 85312 17147 120 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4565] 33 4565 95000 26641 136 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4566] 33 4566 84041 15744 117 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4626] 33 4626 84444 16130 117 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4732] 33 4732 95024 27384 139 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5056] 33 5056 79887 11825 108 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5283] 33 5283 81158 12768 109 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5284] 33 5284 79773 8968 103 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5285] 33 5285 81363 10349 106 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5286] 33 5286 81152 12778 109 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5330] 33 5330 81157 12702 109 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5331] 33 5331 82528 14922 114 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5372] 33 5372 79795 9117 103 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5373] 33 5373 81144 12785 109 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5374] 33 5374 81160 12785 109 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5674] 33 5674 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5676] 33 5676 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5677] 33 5677 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5680] 33 5680 79744 8865 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5681] 33 5681 79746 8868 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5682] 33 5682 79743 8866 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5683] 33 5683 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5688] 33 5688 79741 8849 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5689] 33 5689 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5690] 33 5690 79742 8863 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5691] 33 5691 79743 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5692] 33 5692 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5694] 33 5694 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5698] 33 5698 79742 8849 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5700] 33 5700 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5704] 33 5704 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5705] 33 5705 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5706] 33 5706 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5707] 33 5707 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5708] 33 5708 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5711] 33 5711 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5713] 33 5713 79743 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5715] 33 5715 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5718] 33 5718 79743 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5719] 33 5719 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5722] 33 5722 79806 8864 102 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5725] 33 5725 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5727] 33 5727 79743 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5729] 33 5729 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5731] 33 5731 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5733] 33 5733 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5736] 33 5736 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5737] 33 5737 79745 8869 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5738] 33 5738 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5739] 33 5739 79743 8866 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5740] 33 5740 79743 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5743] 33 5743 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5747] 33 5747 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5749] 33 5749 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5750] 33 5750 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5751] 33 5751 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5754] 33 5754 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5757] 33 5757 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5759] 33 5759 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5761] 33 5761 79806 8849 102 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5762] 33 5762 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5763] 33 5763 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5766] 33 5766 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5769] 33 5769 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5771] 33 5771 79743 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5772] 33 5772 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5775] 33 5775 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5777] 33 5777 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5779] 33 5779 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5780] 33 5780 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5781] 33 5781 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5784] 33 5784 79742 8855 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5787] 33 5787 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5788] 33 5788 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5790] 33 5790 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5793] 33 5793 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5794] 33 5794 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5796] 33 5796 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5800] 33 5800 79744 8866 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5801] 33 5801 79743 8863 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5802] 33 5802 79742 8849 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5803] 33 5803 79746 8868 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5804] 33 5804 79743 8866 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5806] 33 5806 79743 8851 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5808] 33 5808 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5811] 33 5811 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5815] 33 5815 79744 8867 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5816] 33 5816 79743 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5819] 33 5819 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5820] 33 5820 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5822] 33 5822 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5825] 33 5825 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5826] 33 5826 79742 8849 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5828] 33 5828 79744 8850 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5831] 33 5831 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5833] 33 5833 79743 8866 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5836] 33 5836 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5837] 33 5837 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5839] 33 5839 79806 8864 102 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5841] 33 5841 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5842] 33 5842 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5843] 33 5843 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5847] 33 5847 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5848] 33 5848 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5850] 33 5850 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5851] 33 5851 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5855] 33 5855 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5857] 33 5857 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5858] 33 5858 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5859] 33 5859 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5864] 33 5864 79600 8081 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5865] 33 5865 79742 8495 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5866] 33 5866 79742 8556 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5868] 33 5868 79598 8411 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5869] 33 5869 79672 8309 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5873] 33 5873 79600 8370 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5874] 33 5874 79600 8257 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5877] 33 5877 79272 7190 99 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5878] 33 5878 79272 7122 99 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5879] 33 5879 79340 7401 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5881] 33 5881 79272 6968 99 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5885] 33 5885 79600 8260 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5887] 33 5887 79340 7175 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5888] 33 5888 79055 6328 98 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5891] 33 5891 78957 4745 95 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5892] 33 5892 78794 3688 90 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5894] 33 5894 79600 8433 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5896] 33 5896 79022 5810 97 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5898] 33 5898 78598 2088 88 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5899] 33 5899 78308 1476 84 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5901] 33 5901 78498 1910 87 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5902] 33 5902 78533 2140 87 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5905] 33 5905 78308 1538 84 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5908] 33 5908 78308 1470 84 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5909] 33 5909 78308 1470 84 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5910] 33 5910 78267 1453 81 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5912] 0 5912 78255 1444 78 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5913] 33 5913 78259 1446 80 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5914] 33 5914 78259 1446 80 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5915] 33 5915 78271 1462 84 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5916] 33 5916 78265 1451 80 0 0 apache2
Feb 10 15:27:13 mylinode kernel: Out of memory: Kill process 2821 (mysqld) score 224 or sacrifice child