Apache 2.2.22
Prefork
1GB Linode
Running 3 virtualhosts, but only one is "active" at the moment. One has a single HTML page and the other has a web application but Tomcat 7 is turned off.

Other things installed: postgres, Tomcat7 (shut off at the moment), and PHP was installed but I uninstalled it to see if it'd help.

To see how slow Apache is running you can visit www.s2bjj.com yourself. Sometimes it will load but often it won't and if it does many images are not rendered. Sometimes even the CSS file doesn't load.

Basically, Apache was running great but one day I noticed the website above was inaccessible. I run it for a friend who owns a small business and it doesn't receive much traffic at all, however I don't have solid numbers to back this up. It does show up if you Google it so I guess it's a possibility it receives more traffic than I think. However, at least right now I don't think that's the main problem, as the slowness begins immediately after an Apache restart.


What I've tried:

- Uninstalling PHP5 doing "apt-get purge PHP5". I still notice the php5 directory sitting in /etc (I believe) so I'm not sure if it's fully uninstalled. I'm a Linux newb. The site above does not host any files with a php extension, nor does any other virtualhost.

- Playing with apache2.conf. The prefork part uses the recommended settings that Linode provides for web server setup. See below.

Timeout 60

KeepAlive Off

<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 80
MaxRequestsPerChild 3000
</IfModule>

I've also tried putting StartServers, MaxSpareServers, and MaxClients to 8 and putting MaxRequestsPerChild to 0 but this did not help. These were recommendations I found while Googling.

Here's what top looks like currently:

top - 08:07:38 up 12:01, 1 user, load average: 0.52, 0.79, 0.82
Tasks: 195 total, 3 running, 192 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.8%sy, 0.0%ni, 79.9%id, 0.0%wa, 0.0%hi, 0.6%si, 18.4%st
Mem: 1013048k total, 997580k used, 15468k free, 14724k buffers
Swap: 262140k total, 3144k used, 258996k free, 688452k cached


free looks like:

total used free shared buffers cached
Mem: 989 975 13 0 14 674
-/+ buffers/cache: 286 702
Swap: 255 3 252

Apache error log. The MaxClients errors are probably from when I set MaxClients really low. I haven't seen that error since.

[Mon Dec 30 19:27:04 2013] [notice] caught SIGTERM, shutting down
[Mon Dec 30 19:27:05 2013] [notice] Apache/2.2.22 (Ubuntu) configured -- resuming normal operations
[Mon Dec 30 19:27:13 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Tue Dec 31 16:46:54 2013] [notice] Graceful restart requested, doing restart
[Tue Dec 31 16:47:00 2013] [notice] Apache/2.2.22 (Ubuntu) configured -- resuming normal operations
[Tue Dec 31 16:47:02 2013] [notice] caught SIGTERM, shutting down
[Tue Dec 31 16:47:03 2013] [notice] Apache/2.2.22 (Ubuntu) configured -- resuming normal operations
[Tue Dec 31 16:47:04 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Tue Dec 31 16:51:12 2013] [notice] caught SIGTERM, shutting down
[Tue Dec 31 17:05:06 2013] [notice] Apache/2.2.22 (Ubuntu) configured -- resuming normal operations
[Tue Dec 31 17:05:07 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Tue Dec 31 17:10:47 2013] [notice] Graceful restart requested, doing restart
[Tue Dec 31 17:10:53 2013] [notice] Apache/2.2.22 (Ubuntu) configured -- resuming normal operations
[Tue Dec 31 17:10:53 2013] [notice] caught SIGTERM, shutting down
[Tue Dec 31 17:10:55 2013] [notice] Apache/2.2.22 (Ubuntu) configured -- resuming normal operations
[Tue Dec 31 17:10:56 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Tue Dec 31 17:13:01 2013] [notice] caught SIGTERM, shutting down
[Tue Dec 31 17:14:10 2013] [notice] Apache/2.2.22 (Ubuntu) configured -- resuming normal operations
[Tue Dec 31 17:14:11 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Tue Dec 31 17:18:20 2013] [notice] Graceful restart requested, doing restart
[Tue Dec 31 17:18:27 2013] [notice] Apache/2.2.22 (Ubuntu) configured -- resuming normal operations
[Tue Dec 31 17:18:39 2013] [notice] caught SIGTERM, shutting down
[Tue Dec 31 17:18:39 2013] [notice] Apache/2.2.22 (Ubuntu) configured -- resuming normal operations
[Tue Dec 31 17:18:42 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Tue Dec 31 17:18:42 2013] [notice] caught SIGTERM, shutting down
[Tue Dec 31 17:19:48 2013] [notice] Apache/2.2.22 (Ubuntu) configured -- resuming normal operations
[Tue Dec 31 17:19:51 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Tue Dec 31 19:57:00 2013] [notice] Graceful restart requested, doing restart
[Tue Dec 31 19:57:07 2013] [notice] Apache/2.2.22 (Ubuntu) configured -- resuming normal operations
[Tue Dec 31 20:03:25 2013] [notice] caught SIGTERM, shutting down
[Tue Dec 31 20:06:48 2013] [notice] Apache/2.2.22 (Ubuntu) configured -- resuming normal operations

Also, something peripheral that is happening to this but that might be related is I'm getting emails every so often regarding my inbound/outbound traffic.

Your Linode, linode337408, has exceeded the notification threshold (5) for inbound traffic rate by averaging 5.44 Mb/s for the last 2 hours.

Are webcrawlers totally bombarding the website? No way there's enough "customer" traffic to cause this.

I'm at the end of my rope with this so any suggestions or assistance anyone can provide would really help. As a last resort I guess I can just cancel my Linode and give my friend a copy of the site and say "There's many other web hosting options out there. Have at it." but I'd like to just figure it out if possible.

Thanks.

5.44Mb/s is a ton of traffic. Take a look at access.log (or run it through some stats package) and see what the traffic is.

ycs2000,
It sounds like you may be getting unwanted traffic. Check your logs to see if you're getting http traffic. If not, perhaps you're getting unwanted DNS (port 53) or NTP (port 123) traffic. Both of these services can get amplification attacks.


MSJ

Try running nethogs. iptraf and and iotop are useful too, for iotop you might want to supply -n to suppress DNS lookups to reduce output clutter.

Re: access.log file, there seems to be traffic from Google, Bing, etc crawlers and some IP addresses (so I'm definitely getting more traffic than I thought) but nothing seemed like an attack.

I'll check nethogs, iptraf and iotop and report back.

I just ran nethogs, iptraf, and iotop. I'm not that familiar with these tools but nothing appears to be going on that is out of the ordinary. Running an iotop I do see something strange:

2993 be/4 tomcat7 0.00 B/s 10.82 K/s 0.00 % 0.00 % java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logg~/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start

tomcat7 shouldn't be running so I have no idea what's going on here.

If you run tomcat from the repo then it may set up a service, run "sudo initctl list | grep tomcat" and see if it's listed.

On Ubuntu use update-rc.d to change its run state if you don't want it to run when the system starts up.

Tomcat is currently not running for certain but I'm still experiencing slowness. Any other ideas? Now I'm even considering Worker but I'm not sure if I'm a good candidate for it. I still have no idea what's causing the current performance issues so it's hard to tell if Worker would do any good.

Can't connect (tried earlier today same thing).

If you're getting hammered, and can figure out what's causing all that traffic, you've probably been compromised.

Might be time to start fresh, ONLY install/config the services you need for the one site, get that sorted out, then (and only then) add more services.

_________________
Either provide enough details for people to help, or sit back and listen to the crickets chirp.
Security thru obscurity is a myth - and really really annoying.

I second this. If that machine is serving 5Mbs of something and it's not your traffic you have to reinstall the machine, there is no other way to be sure.

For all you know it could be serving kiddie porn.

Unfortunately I think you guys might be right.

The site is inaccessible this morning because I shutdown apache last night to see if that helped with my high/unusual incoming/outgoing. Oddly enough, it did. So somehow Apache is causing super weird and high incoming/outgoing.

In light of the fact that incoming/outgoing traffic went way down after shutting down apache do we still think that wiping my linode is the best option? Are there no other apache config options I can explore?

Do you know WHAT is causing that much traffic?

All you now know is that it's using Apache. So Apache, or one of it's processes is probably compromised?

Personally, my rule of thumb is Four Hours. If I can't figure out the problem in four hours, then it's time to start over. YMMV

At some point, you are in a losing game. Between downtime and your time futzing around trying to fix it, you've exceeded the time it would take to start from scratch.

That's why build logs (i.e. how did you set the server up - exactly) and known good data backups are key.

But remember, all of this is pretty much "blind advice" - you're the man onsite, you're the sysadmin, and you're the only one that can decide what's best for your system.

Perhaps you're not compromised, perhaps your Public IP is poisoned? But without knowing where the traffic is coming from, or what the traffic is, all of this is just a guess.

_________________
Either provide enough details for people to help, or sit back and listen to the crickets chirp.
Security thru obscurity is a myth - and really really annoying.

You bring up a good point. At this point it doesn't appear I'm going to be able to find a solution, and my buddies do need their site up, so I need to just wipe the linode and move on. Unfortunately, my sysadmin skills are severely lacking; I'm a tenderfoot at best. I'm afraid to move forward until I find out what is causing the issues, whether it be a compromise or misconfiguration, because what if this happens in the future. But the good thing about this being a favor and being a tenderfoot is I'm not expected to be an expert so if it happens again I can just re-wipe and move on with my life.

Thanks to everyone who has assisted with this. I appreciate everything.

When I'm off work I'm going to install tcptrack, re-start apache and try to see if I can find what is causing the high traffic. This will be my last task before wiping.