After today's Linode downtime/unplanned outage/ rebooting issues I cannot ssh to my server and none of my websites are reachable. I have reboot the linode and does not help. I tried using the ajax console to see if ssh is running or what but it seems stuck in boot with this message:

...
INIT: /etc/inittab[33]: rlevel field too long (max 11 characters)
INIT: /etc/inittab[34]: rlevel field too long (max 11 characters)
INIT: /etc/inittab[35]: rlevel field too long (max 11 characters)
INIT: /etc/inittab[36]: missing action field
Enter runlevel:

If I enter runlevel 3 I get this error:

INIT: Entering runlevel 3                                         
INIT: no more processes left in this runlevel

Is there anything I can do at this point? I tried submitting a support ticket but I am sure they are swamped. I need to get back up and running asap!!

this is a link to the forum topic regarding linode's unplanned outage:
http://www.linode.com/forums/viewtopic.php?t=4765&postdays=0&postorder=asc&start=0

I'm having pretty much the same problem.

EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 220k freed
INIT: version 2.86 booting
INIT: No inittab file found

Enter runlevel: 1
INIT: Entering runlevel: 1
INIT: no more processes left in this runlevel


I really hope my disks didn't get wiped. I realized the server was down this afternoon when I logged in to run a backup after losing my local data.

can you get in by lish?

id run fsck

i know the usual place for this information is in the forum/irc but a separate out-of-band place to get this information would be good (as others have suggested).

A suggestion would be a wordpress blog on an off-site host. so that even if the linode website is down, there is still a way to get information out to your customers.

You could use a wordpress plugin to let people subscribe to email notifications of new posts.

Interested parties could subscribe to an email list to get outage information 'pushed' to them, instead of relying on checking an rss feed or forum post for updates.

The box doesn't boot, I could fsck all day and it won't help, it doesn't appear to have a ram disk to load.

But:

root@hvc0:/# fsck /dev/xvde
fsck 1.40.6 (09-Feb-2008)
e2fsck 1.40.6 (09-Feb-2008)
Adding dirhash hint to filesystem.

/: clean, 59802/652800 files, 545813/640000 blocks



I can mount the rescue image, and when I mount my root partition, I have no inittab in /etc, and I can't recreate it, either.

I do see this if I do an ls -l on my /etc directory, though, but not if I run 'ls' with no flag:

?--------- ? ? ? ? ? inittab

Which is kind of weird. And if I try to list it alone:

root@hvc0:/2/etc# ls -l inittab
ls: inittab: No such file or directory

Twitter works for me as a place to lodge brief status messages: http://twitter.com/linode

This might help: http://www.linode.com/forums/viewtopic.php?t=4769

Unfortunately, this doesn't work for me.
I can't modify my /etc/inittab file. When I try to, it tells me that the file isn't there. If I try to create a new one, it tells me that the file isn't there. I'm not exactly sure what the problem is.

This was due to /etc/inittab, /etc/fstab, and a few other files getting eaten by an odd bug that cropped up due to our outage. If you are experiencing this, please file a ticket and we can have you fixed in a few minutes.

We went through and fixed all the Linodes we could find that were suffering from this (by grepping your boot logs and looking for common symptoms), but there is the chance we missed some. I encourage you to file a quick ticket, we'll shut you down, do some magic on the host, boot you up, and get you on your way.

Sorry for all the problems that Tuesday's outage caused.

_________________
Disclaimer: I am no longer employed by Linode; opinions are my own alone.