Now that Xen is getting rolled out and VMs can be allowed into their own kernel space with modules and so forth, I think stock distro kernels would be a useful intermediate step to allowing people to do custom kernels.

With custom kernels the Linode host would have to safely extract the image from a VM's filesystem in order to load it with Xen. However, with stock distro kernels, only the existing functionality already provided by the Linode host would be required to make it work, it would simply be an extra choice in the list that's already there. Many distros provide Xen kernels, or kernels that support paravirt_ops, so Linode staff could obtain the appropriate kernel images and associated initrd's by simply running that distro inside a VM and keeping it up to date with the local package manager.

This would not allow for fully custom kernels, but it would eliminate problems caused by a Linode's kernel and its modules/userspace tools not being in sync.

Why not just rebuild the userland tools as needed against your running kernel? Right now, Linode offers 14 distributions from the distro wizard, so that would mean that they would have to follow at least 14 different kernels. Plus, this opens the door for people with custom distros to want their kernels included too, etc.

Avoiding the need to do custom compiles is a large part of the motivation to using distros like Debian/CentOS/etc that provide binaries. Particularly on multiple machines, custom compiles add significantly to maintenance overhead.
It would not be required that all be directly supported. Eg, it would be a good compromise to only support major distros that provide suitable kernels easily. This approach is not mutually exclusive with the current one, though if all distros were supported it would remove the need for Linode to maintain its own kernel.
It has been mentioned that custom kernels are not out of the question in the future, but that is significantly more complex because Xen needs to be able to safely read the kernel into memory from the guest's filesystem. I would say that the door is opened by the move away from UML, that changes a lot of things.

Can you elaborate which userspace tools are incompatible with the kernels we provide? I've never heard of such a problem before, so I'm interested to know.

Thanks,
-Chris

Sure.

I recently had an issue with cryptsetup that was entirely my fault ( ), but another thing I noticed after the private network was set up was that ocfs2/drbd support was not yet available.

With a Linode Xen kernel that allowed the loading of modules this could be circumvented, but it would still be necessary to maintain one's own modules from source as the distro-provided modules would not match.

EDIT: hm... it looks like the cryptsetup stuff was partially attributable unexpected differences between userspace and the kernel. Ubuntu's init scripts expect the dm and crypto stuff to be modules. They're supported, but they're not modules. The script has to be modified not to attempt to load them, or the system will go into single-user mode on the next boot.