Ok, so what started out as a simple problem has turned in to a catastrophic failure. I started getting emails from a cron job complaining about:

Error: Write failed : Bad file descriptor

Uh oh. So I run fsck, and it starts complaining big time. So before I reboot to single user mode, I jump on the lish console, and it's being flooded with messages like this:

EXT3-fs error (device ubda) in ext3_reserve_inode_write: IO failure
EXT3-fs error (device ubda): ext3_get_inode_loc: unable to read inode block - inode=14499, block=33554435
EXT3-fs error (device ubda) in ext3_reserve_inode_write: IO failure
EXT3-fs error (device ubda): read_block_bitmap: Cannot read block bitmap - block_group = 0, block_bitmap = 2553887680
EXT3-fs error (device ubda): read_block_bitmap: Cannot read block bitmap - block_group = 0, block_bitmap = 2553887680

I bring my system down via a hard stop. Even /etc/shutdown won't work. It gave me a permission denied error.

When I try and boot back up the linode, I get the following error in the job queue:

helper_main(petercrocker, /dev/vg1/petercrocker-44280, 1, , 1): mount failed: File exists

I've searched the forums, and the one other hit I see doesn't seem relevant.

So I setup a finnix recovery profile, boot it, and run fsck on my linode, and it looks like it's not going to fix anything:

root@tty0:/sbin# fsck /dev/ubdb
fsck 1.39-WIP (31-Dec-2005)
e2fsck 1.39-WIP (31-Dec-2005)
Group descriptors look bad... trying backup blocks...
ext3 recovery flag is clear, but journal has data.
Recovery flag not set in backup superblock, so running journal anyway.
/dev/ubdb: recovering journal
Block bitmap for group 0 is not in group. (block 2553887680)
Relocate<y>? yes

Inode bitmap for group 0 is not in group. (block 16777216)
Relocate<y>? yes

Inode table for group 0 is not in group. (block 1949373184)
WARNING: SEVERE DATA LOSS POSSIBLE.
Relocate<y>? yes

Eh. Not good.

So if any of you can offer advice, I've got the following questions:

1) Know a way to repair in a case like this?

2) Could a resize operation have caused this?

3) Assuming this disk is toasted, then ideally I'd like to resize the image. It's about 4 gigs now, but I made it a 10 gig image. I'd like to be able to salvage some files off of it by creating a new linode image, and mounting the old image on some mount point. But I can't resize, because the linode resizing tool is smart enough to know there's disk corruption:

fs_resize - resize failed

resize2fs 1.38 (30-Jun-2005)
Please run 'e2fsck -f /dev/vg1/petercrocker-44280' first.

Anyone got creative suggestions on how to accomplish this, or should I just pony up and order some more drive space?

/dev/vg1/petercrocker-44280 is an LVM logical volume on the host.

Raise a ticket or grab caker/mikegrb/tasaro in the IRC channel and see if they can fix it.

_________________
/ Peter