Hi,

Recently, I have had two spikes in traffic to my website. Average connections per second have been about 180. These spikes were at about 600 and 800 connections per second. There seemed to be some funny business going on after looking at the server logs of the content that was served; likely scraping content from my website.

I was wondering have any of you experienced this. How did you or would you deal with it? Do any of you have experience with investigating and taking action other than beefing up servers (maybe legal action)?

My server configuration has been as follows:
NodeBalancer with two 1GB web servers (serving HTTP on port 80 and HTTPS on port 443) running Apache. These servers connect to one 2GB database server running MySQL.

I am migrating to two 2GB SSD web servers running LiteSpeed and optimizing my web app to server more static files from a CDN to reduce overall connections.

So, again. Does anyone have experience and recommendations for handling this load? Anyone have experience with researching this kind of activity to determine who is inadvertently knocking down my site?

Let me know if you need more information.

Thanks, Josh

Not that it's much consolation, but I doubt they (whomever 'they' are) were trying to knock down your site. This game of 'block the scrapers' is always going to be pretty much reactionary unless you start blocking large segments of IPs in advance.

When we see this type of activity we block the entire range that owns the offending IP addresses unless it appears to be a consumer/residential IP address. It's not perfect, but it prevents return visits and gives us at least a little bit of a sense of doing something about it.

I had the same problem (to a lesser extent), and used ModSecurity along with the OWASP Core Ruleset to deal with it. Modsecurity is a pain to setup, but it's free and very powerful. The OWASP ruleset has a ton of rules to deal with bots, and they're pretty good at keeping it up-to-date.

You might look into setting up Cloudflare in front of your site. With it's CDN and caching and WAF rules you'd probably be all set without having to install anything locally...

I HIGHLY suggest cloudflare and then varnish in front of the web server. We run 4 servers, 1 with varnish 2 apps and a database server. We can handle some really massive spikes with minimal impact.