Linode Forum

Xan · Joined: 08 Feb 2004 Posts: 562 Location: Austin

http://www.theregister.co.uk/2009/06/08/webhost_attack/

An ugly, ugly story, capped by the suicide of the owner/lead developer.

How glad I am that I'm with a provider whose developer is infinitely more clueful! Thanks Caker.

waldo · Joined: 21 May 2009 Posts: 336

It'll never end. People who have absolutely no business creating software, managing hardware, or managing software will always, create software, manage hardware or manage the software.

There was a local "host" who was mostly just a crew of web designers who were renting 2 servers in a colo. They never backed anything up, even from their local workstations. They thought a set of mirrored drives in the server was sufficient. The file system on one of the servers became corrupt and they were hosed. They decided to just shut their doors as they realized, after screwing over 3/4 of their customers, they didn't have the know how to manage servers or web sites, just create them. Granted they were able to recover a lot of their data because it was laying around on workstations and such.

Quote: Some 50 percent of Vaserv's customers signed up for unmanaged service, which doesn't include data backup, Foster said. It remains unclear of those website owners will ever be able to retrieve their lost data, he said.

This is why, even at a quality host, I always backup my own data and always urge others to do the same. You and only you are responsible for your data.

Xan · Joined: 08 Feb 2004 Posts: 562 Location: Austin

waldo wrote: This is why, even at a quality host, I always backup my own data and always urge others to do the same. You and only you are responsible for your data.

Well put. Every story like this is a message to the whole world: data loss is coming for you. Be backed up.

hybinet · Joined: 02 May 2008 Posts: 1058

Been watching this story unfold, too. Very sad.

I've been with VAServ a couple of times for little projects. Their service was pretty good, especially for the price. (Compared to other "ordinary" hosts, of course. Linode isn't an "ordinary" host!) Too bad they suffered this massive outage due to a piece of software they had virtually no control over.

VAServ seems to be recovering pretty well, at least for now. I hope they survive this mess. Since they have a pretty loyal customer base, hopefully it wouldn't be too difficult. On the other hand, the Indian guy who developed/sold HyperVM (the control panel software which is deemed responsible for the vulnerability) committed suicide, and a lot of other "ordinary" hosts have been using it too... so there's definitely a storm coming over there.

Lessons

1. Use free and open-source software whenever you can. Obfuscated PHP to prevent copying? Gimme a break.

2. If you create an in-house solution instead, do it well (like Linode!)

hybinet · Joined: 02 May 2008 Posts: 1058

waldo wrote: This is why, even at a quality host, I always backup my own data and always urge others to do the same. You and only you are responsible for your data.

Yup.

And never rely only on your webhost's backup service, not even at Linode. What good is a backup if it's in the same datacenter? Backup, backup again, backup your backups, and backup the backup of your backups!

waldo · Joined: 21 May 2009 Posts: 336

hybinet wrote: And never rely only on your webhost's backup service, not even at Linode. What good is a backup if it's in the same datacenter? Backup, backup again, backup your backups, and backup the backup of your backups!

And another step a lot of people forget, test your backups. What good is a backup if it's corrupt or contains corrupt data?

I knew someone who was "backuping" their database on a regular basis, even taking the backups off site. Everything looked great. Until one day they needed to recover, turns out because of some file-locking issue the database hadn't actually been backed up....

They had only tested when the system was first put into place, everything worked and looked great. At some point over the years something stopped working, but they didn't even do yearly tests, let alone quarterly, monthly or weekly.

Fortunately for them the data they needed was for research purposes and not to recover because of failure.

OverlordQ · Joined: 04 Jun 2004 Posts: 312

Any of the other bigish VPS providers around use this to much?

sweh · Joined: 13 Apr 2004 Posts: 561

hybinet wrote: Lessons

1. Use free and open-source software whenever you can. Obfuscated PHP to prevent copying? Gimme a break.

2. If you create an in-house solution instead, do it well (like Linode!)
3. Don't use a provider that knowingly aided and abetted phishers and other types.

http://tacit.livejournal.com/297618.html

http://tacit.livejournal.com/299317.html

tim101 · Joined: 09 Jun 2009 Posts: 2

Blergh!

I used to host at linode and switched to fsckvps in January? because it saved me $10/month.

...
Hey look, I'm back, not much richer and tons more frustrated.

Luckily for me, my data was saved.

Ughhhhh :(.

hybinet · Joined: 02 May 2008 Posts: 1058

tim101 wrote: I used to host at linode and switched to fsckvps in January? because it saved me $10/month.
You Get What You Pay For (tm)

Honestly though, fsckvps was pretty good for $9.95/mo, at least when compared to other sub-$10 providers (of which there are more than a handful).

The question is: Do you want $9.95 worth of service, or do you want $19.95 worth of service? There's a good reason why Linode refuses to offer anything below that price point, even though there may very well be a market for something like Linode 180.

tim101 · Joined: 09 Jun 2009 Posts: 2

hybinet wrote: Honestly though, fsckvps was pretty good for $9.95/mo, at least when compared to other sub-$10 providers (of which there are more than a handful). Quite correct. Really, to be honest, fsck was great until the HyperVM attack. :(
HyperVM itself was inferior to Linode's control panel- but really, I didn't have to mess with it very much.
Quote: The question is: Do you want $9.95 worth of service, or do you want $19.95 worth of service? I have learned my lesson, as I bet many others have too.

It took a good 6-7 hours work after 48 hours downtime to get everything online over here. I had backups but they were not recent (another lesson learned!-I guess if the worst happened I would have had something though). In the end I had to wait for them to get http online and hack (irony time) a script that I made to run the command through php to create a tar of my files and transfer here. Ssh (so sftp) is still offline, so I couldn't download files the normal way. This was on my server anyway, they are not really responding to tickets.

I feel for less technical fsck-ers & those who lost data.

Anyway, I'm glad to be live & back at linode :wink:

neo · Joined: 01 May 2009 Posts: 83

waldo wrote: And another step a lot of people forget, test your backups. What good is a backup if it's corrupt or contains corrupt data?

I knew someone who was "backuping" their database on a regular basis, even taking the backups off site. Everything looked great. Until one day they needed to recover, turns out because of some file-locking issue the database hadn't actually been backed up....

They had only tested when the system was first put into place, everything worked and looked great. At some point over the years something stopped working, but they didn't even do yearly tests, let alone quarterly, monthly or weekly.

Fortunately for them the data they needed was for research purposes and not to recover because of failure.
Yes, backing up is simple, it's the restoring which tends to get hard... Remember an old joke about announcement of revolutionary compression software able to compress any data to 100 bytes. Now developers started work on the decompression part...

josephb · Joined: 20 May 2008 Posts: 17

tim101 wrote:
It took a good 6-7 hours work after 48 hours downtime to get everything online over here.

When you add up how much that is worth to you in lost time/revenue etc the $10 a month saving would be blown away for a few years :)

Good to see you back!

hybinet · Joined: 02 May 2008 Posts: 1058

neo wrote: Remember an old joke about announcement of revolutionary compression software able to compress any data to 100 bytes. Now developers started work on the decompression part...

LOL :D

Well, it ain't just a joke. MD5 can "compress" anything to 16 bytes, and it's considered "weak" so theoretically it should be possible to "decompress" the hash to its original representation... Just bruteforce it for a few weeks and look for a collision!

Silver Blade · Joined: 30 Jun 2007 Posts: 5 Location: Oxfordshire, UK

Yeesh, nasty... I have a very basic VPS based with vaserv which I thankfully don't use for anything important.

I'm glad all my other stuff is on Linode. I really ought to get that backup script working again though...

Edit: Just remembered, I had considered applying for a job vacancy they had at a2b2/cheapvps previously (part of vaserv.) Probably a good thing I didn't!

SQL injection brings down VPS host