All of a sudden one of my mounted partitions became read-only and was unable to create files even though it's mounted rw and has been working fine for quite some time. Had to reboot for it to come back rw.

scratch # df .
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvdc 18031711 7943015 10088696 45% /scratch
scratch # mount
/dev/xvda on / type ext3 (rw)
none on /dev type ramfs (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw)
none on /dev/shm type tmpfs (rw)
/dev/xvdc on /scratch type ext3 (rw)
scratch # touch foo
touch: cannot touch `foo': Read-only file system
scratch #
scratch #
scratch # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda 2.0G 779M 1.3G 39% /
none 361M 0 361M 0% /dev/shm
/dev/xvdc 18G 7.6G 9.7G 45% /scratch

This is a Xen node. How can this happen?

I actually found the following on Lish:

EXT3-fs error (device xvdc): ext3_readdir: bad entry in directory #1127702: directory entry across blocks - offset=0, inode=3970859677, rec_len=6624, name_len=248
EXT3-fs error (device xvdc): ext3_readdir: bad entry in directory #1127702: directory entry across blocks - offset=0, inode=3970859677, rec_len=6624, name_len=248
ext3_abort called.
EXT3-fs error (device xvdc): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only

Since I've converted to Xen I've been having numerous problems. How is Xen working for others?

Xen has this bug where the FS can break if you're on 1K-size blocks. You'll need a 4k-block-size FS. I'm not sure if you can do that everywhere (if it's the default for new filesystems everywhere yet), so open a ticket or wait for a staff reply here.

This is caused by a bug in Xen with filesystems formatted with 1k blocksize. We're still trying to reproduce, so any hints would be appreciated.

This can solve this by moving the data to a filesystem formatted with 4k blocksize. The Linode Manager now creates all new disk images with 4k blocksize, and all of our distribution templates have been converted to 4k bs (if they weren't already). Submit a ticket and we can provide your Linode with some temporary space while you make the data move, or we'd be happy to do it for you.

-Chris

Since I had to reboot I'm currently resizing that partition to pick up the extra disk space you graciously gave us. I assume that resize job is converting to the 4k block size?

Resizing does not change the blocksize. You can check an ext2/3 filesystem's blocksize like so:

tune2fs -l /dev/whatever | grep "Block size"

-Chris

Can you paste the line from /etc/fstab for this file system? I'm interested in the specific options the fs is mounted with.

Mike

/scratch was the partition that failed and is using a 1k block size.


~ # tune2fs -l /dev/xvdc | grep "Block size"
Block size: 1024
~ # cat /etc/fstab
#
# /etc/fstab: static file system information
#
# <file system> <dir> <type> <options> <dump> <pass>
none /dev/pts devpts defaults 0 0
none /dev/shm tmpfs defaults 0 0

/dev/xvda / ext3 defaults 0 1
/dev/xvdb swap swap defaults 0 0
/dev/xvdc /scratch ext3 defaults 0 0

Can you try

debugfs /dev/xvdc

and then

dump <3970859677> dumpfile
stat <3970859677>
cat <3970859677>

And then email dumpfile plus the output of the above to mgreb@linode.com?

It appears the inode is gone after I resized the partition during the reboot...

# debugfs /dev/xvdc
debugfs 1.40.8 (13-Mar-2008)
debugfs: dump <3970859677> dumpfile
dump: Illegal inode number while reading inode 3970859677
debugfs: quit

If it happens again I'll get you this information.

right forgot about the resize, thanks though